Machine learning algorithms for detecting medical conditions, related systems, and related methods

ABSTRACT

Systems for preparing, training, and deploying a machine learning algorithm for making medical condition state determinations include at least one processing unit that includes the machine learning algorithm. The at least one processing unit is programmed to receive image input from an imaging device, receive patient health data, encode the patient health data to convert the patient health data to encoded patient health data, and transmit the encoded patient health data into the machine learning algorithm. Systems are configured to make a medical condition state determination based on the image input and the encoded patient health data, via the machine learning algorithm, and provide visual output for the medical condition state determination via a display device such that the visual output may be augmented with the patient health data. Dynamic state information also may be input to the machine learning algorithm and used to make medical condition state determinations.

RELATED APPLICATION

This application claims priority to U.S. Provisional Patent ApplicationNo. 63/041,527, filed on Jun. 19, 2020, and entitled MACHINE LEARNINGMODEL FOR DETECTING MEDICAL CONDITIONS.

FIELD

The present disclosure relates generally to machine learning algorithmsfor detecting medical conditions, and more particularly to machinelearning algorithms, systems, and methods for real-time analysis ofmedical images from medical imaging procedures.

BACKGROUND

Medical imaging is widely used for screening and diagnosis of a widevariety of medical conditions, and may include techniques such assonography videos, x-ray films, computed tomography (CT) scans, magneticresonance imaging (MRI) scans, positron emission tomography (PET) scans,retinal photography, histology slides, dermoscopy images, radiography,mammography, as well as laparoscopic videos, endoscopic techniques,including lower endoscopy (e.g., colonoscopy), upper endoscopy (e.g.,esophagogastroduodenoscopy), bronchoscopy, and capsule endoscopyprocedures (e.g., Pillcam™) for examining the entire digestive system.Such visual inspections can be used in screening or diagnosing cancer,lesions, auto-immune diseases, infections, and many other medicalconditions. Images (or in some cases, videos) produced via these andother medical imaging procedures can be too numerous for examiningphysicians to individually analyze each image. Artificial intelligenceis thus increasingly utilized in analyzing and interpreting images frommedical imaging procedures.

For example, machine learning models such as convolutional neuralnetworks (CNNs) have been trained to analyze medical images and performclassification and diagnoses of various conditions. Such models havebeen trained using datasets having a feature of interest and datasetsthat do not include the feature to “learn” a function. Once the model istrained and validated, it can then be used to make determinations on newdata/inputs, and thus aid health care workers in medical image analysis.In a specific example, CNNs have been trained to detect and localize alesion in a colonoscopy procedure. While such models have improved inaccuracy in recent years, many remain limited in the speed at which theycan analyze medical images. Existing models also are limited in theirability to present visualization of data from multiple data sources tothe examining physician, in their training methods, and/or in theirability to receive larger image datasets for analysis.

SUMMARY

Presently disclosed systems for preparing, training, and deploying amachine learning algorithm for medical condition state determinationinclude at least one processing unit comprising the machine learningalgorithm. The at least one processing unit may be programmed to receivean image input, receive patient health data as input, encode the patienthealth data to convert the patient health data to encoded patient healthdata, transmit the encoded patient health data into the machine learningalgorithm, and make a medical condition state determination based on theimage input and the encoded patient health data, via the machinelearning algorithm. The image input generally includes one or moreimages from an imaging device, such as from a colonoscopy or othermedical imaging procedure.

Presently disclosed methods of training and preparing a machine learningalgorithm for medical condition state determination may includeacquiring data from at least one medical procedure. For example,acquiring data may include acquiring at least one in situ biologicalimage of an area of a body of a patient and/or acquiring one or morebiological specimens from the area. Methods also may include labelingthe at least one in situ biological image, thereby creating at least onelabeled biological image that indicates respective medical conditionstates shown in each respective biological image, acquiring patienthealth data pertaining to the patient from a plurality of data sources,and aggregating the patient health data acquired from the plurality ofdata sources into a database (e.g., a text-based or other form ofdatabase). The patient health data in the database may be de-identifiedin some methods. In this manner, methods may include training themachine learning algorithm using the data from the database and the atleast one labeled biological image.

In other presently disclosed methods of training and preparing a machinelearning algorithm for making a medical condition state determination,the method may include receiving an image input via at least oneprocessing unit, wherein the image input comprises one or more imagesfrom an imaging device, and wherein the at least one processing unitcomprises a machine learning algorithm, and receiving patient healthdata as input, wherein the receiving patient health data is performed bythe at least one processing unit. Such methods may further includeencoding the patient health data and thereby converting the patienthealth data to encoded patient health data, wherein the encoding thepatient health data and the converting the patient health data isperformed by the at least one processing unit. Disclosed methods alsomay include embedding the encoded patient health data into at least oneimage of the image input, wherein the embedding the encoded patienthealth data is performed by the at least one processing unit, whereinthe machine learning algorithm is configured to make the medicalcondition state determination based on the image input and the encodedpatient health data. Other related systems and methods also aredisclosed, along with the machine learning algorithms themselves.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic representation of non-exclusive examples ofsystems for detecting medical conditions, according to the presentdisclosure.

FIG. 2 is another schematic representation of non-exclusive examples ofsystems for detecting medical conditions, according to the presentdisclosure.

FIG. 3 is an example of an output image produced by presently disclosedsystems.

FIG. 4 is an example of an output image produced by presently disclosedsystems.

FIG. 5 is an example of an output image produced by presently disclosedsystems.

FIG. 6 is an example of an output image produced by presently disclosedsystems.

FIG. 7 is an example of an image produced by presently disclosedsystems, with embedded encoded patient health data displayed on theimage.

FIG. 8 is an example of an output image including visual outputaugmented by patient health data according to the present disclosure.

FIG. 9 is a high-level schematic flowchart of methods of preparing andtraining machine learning algorithms according to the presentdisclosure.

FIG. 10 is a schematic flowchart representation of methods of acquiringand/or collecting data for training disclosed machine learningalgorithms.

FIG. 11 is a schematic flowchart representation of methods of labelingdata for training and preparing disclosed machine learning algorithms.

FIG. 12 is a schematic flowchart representation of methods ofaggregating and preparing data for training disclosed machine learningalgorithms.

FIG. 13 is a schematic flowchart representation of methods ofde-identifying and preparing data for training disclosed machinelearning algorithms.

FIG. 14 is a schematic flowchart representation of methods of trainingand deploying presently disclosed machine learning algorithms for makingmedical condition state determinations.

FIG. 15 is a schematic flowchart representation of methods of trainingand deploying presently disclosed machine learning algorithms for makingmedical condition state determinations.

FIG. 16 is a schematic representation of non-exclusive examples ofmachine learning algorithm architectures that may be employed inpresently disclosed systems.

FIG. 17 is a schematic representation of non-exclusive examples ofmachine learning algorithm architectures that may be employed inpresently disclosed systems.

FIG. 18 is another schematic representation of non-exclusive examples ofmachine learning algorithm architectures that may be employed inpresently disclosed systems.

DESCRIPTION

Systems according to the present disclosure may be used to prepare,train, and deploy machine learning algorithms for medical conditionstate determinations.

FIGS. 1-2 provide illustrative, non-exclusive examples of systems 10according to the present disclosure. Elements that serve a similar, orat least substantially similar, purpose are labeled with like numbers ineach of FIGS. 1-2, and these elements may not be discussed in detailherein with reference to each of FIGS. 1-2. Similarly, all elements maynot be labeled in each of FIGS. 1-2, but reference numerals associatedtherewith may be utilized herein for consistency. Elements, components,and/or features that are discussed herein with reference to one or moreof FIGS. 1-2 may be included in and/or utilized with any of FIGS. 1-2without departing from the scope of the present disclosure.

In general, elements that are likely to be included in a given (i.e., aparticular) example are illustrated in solid lines, while elements thatare optional to a given example are illustrated in dashed lines.However, elements that are shown in solid lines are not essential to allexamples, and an element shown in solid lines may be omitted from aparticular example without departing from the scope of the presentdisclosure.

FIG. 1 schematically illustrates nonexclusive examples of systems 10according to the present disclosure, showing systems 10 at a high level,overall perspective. Systems 10 include at least one processing unit 12which includes a machine learning algorithm 11. In some examples,machine learning algorithm 11 may be accessed by processing unit 12,rather than stored therein. Processing unit 12 is programmed to receivean image input 14 from an imaging device 16, with image input 14 beingone or more images, slides, and/or videos that are obtained and/orproduced by imaging device 16. For example, image input 14 may includedigitized pathology slides, videos, CT images, or any other type ofimage produced by imaging device 16. Imaging device 16 may be, forexample, a sonography device, an x-ray device, a computed tomography(CT) scanning device, a magnetic resonance imaging (MRI) device, apositron emission tomography (PET) device, a retinal camera, adermatoscope, a radiograph device, a mammography device, an endoscope, acolonoscopy device, an esophagogastroduodenoscopy device, a bronchoscopydevice, a photoacoustic endoscopy device, an electro-optical sensor, aNarrow Band Imaging (NBI) colonoscopy device, a white light endoscopydevice, a chromoendoscopy device, and/or a capsule endoscopy device.Imaging device 16 may be located in the same facility as processing unit12 in some examples. In other examples, imaging device 16 may be locatedin a different facility and/or location than processing unit 12. Forexample, one or more remotely located imaging devices 16 may send imageinput 14 to processing unit 12, such that machine learning algorithm 11may be used to process and analyze data from a plurality of differentimaging devices 16 concerning a plurality of different patients.

Processing unit 12 is further programmed to receive patient health data18, which may also be known as meta-data 18, as input. Patient healthdata 18 may be received from, for example, a computing device 28, whichmay store said patient health data 18 and/or be configured to accesspatient health data 18 from other sources such as manual input,electronic health records, electronic medical records, and/or otherhealth/medical records or charts. Patient health data 18 may includepatient health data that is entered in real-time (e.g., during orimmediately before or after a procedure producing images via imagingdevice 16), patient health data from said medical records, dynamic statedata (real-time, potentially continuously changing data), and/or staticdata regarding the patient. In some examples, patient health data 18includes data that is collected or provided before image input 14 isprovided to processing unit 12. For example, processing unit 12 mayreceive patient health data 18 that includes dynamic heart rate datasynced with a colonoscopy procedure, along with patient's demographicsaccessed from medical records, along with information entered the day ofthe colonoscopy, such as information pertaining to the procedureindication, previous colonoscopy results and preparation quality,medications, and time since last food or drink consumed. As furtherillustrative examples, patient health data 18 may include surveyquestion answers, static data, active data, electronic health records,electronic medical records, risk factors, body mass index (BMI),physical activity, cigarette smoking history, alcohol usage, familyhistory, presence of inflammatory bowel disease, current hormone therapy(e.g., postmenopausal hormone therapy), former hormone therapy (e.g.,postmenopausal hormone therapy), aspirin usage, nonsteroidalanti-inflammatory drugs (NSAIDs) usage, consumption of processed and/orred meat, fruit and vegetable consumption levels, demographicinformation, medications (e.g., aspirin, folate, multivitamins,prescription medications, etc.), drug usage, diet type and quality,dietary fat intake, weight, height, age, race, presence of otherillnesses or diseases (e.g., Lynch syndrome, IBS, hereditarynonpolyposis, colorectal cancer, diabetes), biological markers (e.g.,markers that may correlate with polyps or colon cancer or other medicalcondition being tested for, such as hemoglobin level, albumin, totalprotein, hemoglobin a1c, creatinine clearance, bilirubin, cholesterolprofile, differential of white blood cells, c-reactive protein, and/orothers), international normalized ratio (INR) test results, partialthromboplastin time (PTT) test results, prothrombin time (PT) testresults, heart ejection fraction, platelet count, bleed time, previousendoscopy results, previous CT scan results, previous angiograminformation, previous MRI results, previous PET results, computed riskpredictors, blood work, prior procedural results, ASCVD risk, liverfailure factors, autoimmune risk factors, Fong Clinical Risk Score forColorectal Cancer Recurrence, and/or previous sonography (e.g.,ultrasound) data. Results from previous tests or procedures can indicatepreviously-identified high-risk areas and/or areas that were poorlyvisualized, such that these areas can be given special attention duringthe current medical imaging procedure. Additionally or alternatively,patient health data pertaining to previous tests or procedures canindicate areas of increased signal intensity, such as in the case ofprior CT scans, MRIs, and/or sonography. In some examples, results fromprevious tests or procedures can give location information about priorsurgeries or treatments, such as the location of a previous polypectomyperformed on that patient. In a specific example, patient health data 18may include information regarding a patient's cardiac cycle and/orbreathing cycle, with image input 14 being temporally annotated suchthat each of one or more images 30 from imaging device 16 may be matchedwith a phase of the patient's cardiac cycle and/or breathing cycle. Suchtemporal annotation (e.g., time stamps) may be relative (e.g.,chronologically ordering each image 30 with respect to the other images30 if image input 14) and/or absolute (e.g., mapped to the specific timeof day that each image 30 was taken).

Systems 10 provide advantages over the prior art in how they canaggregate and combine temporally dissimilar data sources (e.g.,real-time image data, real-time dynamic state information, and/orpreviously collected patient health data) to infer or determine amedical condition state determination in real-time. This may beaccomplished both by integrating the data into a data tensor prior toinput into machine learning algorithm 11, and/or optionally integratingthe data into a data tensor for usage in machine learning algorithm 11.In some examples, previously collected patient health data 18 may beabstracted, or converted, into a numerical data representation (e.g., adata tensor or vector) that is conducive to combining with real-timesensor data and/or with image input 14 that has been converted to tensordata within machine learning algorithm 11. This may enable systems 10 toutilize a single data representation that can be acted upon by machinelearning algorithm 11 by combining these multiple sources and types ofdata together. Disclosed machine learning algorithms 11 may exhibitimproved accuracy and/or availability in medical condition statedeterminations, as compared to human diagnosis and/or prior artartificial intelligence diagnoses, due to the elimination ofsubjectivity and the introduction of objective decision-making processesbased on training of machine learning algorithm 11. Prior art machinelearning models are not equipped to incorporate such patient health datafrom different points in time and have less access to data, and thus areless capable of evaluating the patient as a whole.

To this end, processing unit 12 is programmed to encode patient healthdata 18 thereby to transform, or convert, patient health data 18 toencoded patient health data 20, and transmit encoded patient health data20 into machine learning algorithm 11 (e.g., feed, transfer, digitallytransfer, deliver, or transport encoded patient health data 20 to, orinto, machine learning algorithm 11). Said encoded patient health data20 may be in the form of a data vector or tensor, and/or in the form ofcoded image pixels, as will be described in more detail herein. Based onimage input 14 and encoded patient health data 20, processing unit 12 isprogrammed to make a medical condition state determination via machinelearning algorithm 11. In an illustrative, non-limiting example, imageinput 14 may be images and/or video footage from a colonoscopy procedureobtained from imaging device 16 in the form of a colonoscopy device.Machine learning algorithm 11 may be trained to analyze this image input14, along with encoded patient health data 20, to make a medicalcondition state determination, such as analyzing whether any of theimages from the colonoscopy device show cancerous polyps.

Machine learning algorithm 11 is a convolutional neural network in someexamples, which may be a fully trained convolutional neural network or atransfer learning convolutional neural network. In some examples,machine learning algorithm 11 is a custom convolutional neural network,while in other examples, machine learning algorithm 11 may employ astandard or generic convolutional neural network. Machine learningalgorithm 11 typically is between 1 and 100 convolutional layers deep,though additional convolutional layers may be used in various examples.The number of fully connected layers of machine learning algorithm 11also may be varied, such as to optimize the configuration of machinelearning algorithm 11 such that it is adapted for a particular task orapplication. Some specific examples of disclosed machine learningalgorithms include a total of between 5 and 20 layers. Additionally oralternatively, machine learning algorithm may be, or include, atransformer, a long short-term memory (LSTM), a recurrent neural network(RNN), a support vector machine (SVM), a dense neural network, anauto-encoder, and/or a vision transformer.

As described in more detail in connection with FIGS. 9-15, machinelearning algorithm 11 may be trained and/or deployed according topresently disclosed methods. In some examples, machine learningalgorithm 11 may be trained using datasets with labels created byunsupervised labeling (which may also be referred to as auto-labeling),semi-supervised labeling, and/or supervised labeling, or manuallabeling. In some examples, supervised labeling is performed beforesemi-supervised and/or unsupervised labeling. In some examples,supervised labeling and unsupervised labeling initially are performed totrain machine learning algorithm 11, with semi-supervised labeling beingperformed as needed or desired to improve and/or refine labels in thedataset. As used herein, “supervised labeling” (or “manual labeling”)refers to labels that are assigned to images by a human,“semi-supervised labeling” refers to a process where a human labels aplurality of images (often a few hundred images), training a rudimentarymachine learning algorithm using these labeled images, and then lettingthe trained algorithm make a guess at the labels for the rest of thethousands of images, then the human verifies that all thoseautomatically assigned labels by the algorithm are indeed correct, and“unsupervised labeling” refers to automated algorithms for labelingtraining images without human supervision.

In some systems 10, processing unit 12 is further configured to receivesensor data 22, such as sound input, a near-infrared (NIR) spectroscopyinput, 2D vector data, 3D vector data, and/or 4D vector data. Forexample, sensor data 22 may include sound input from a health carepractitioner dictating information about the procedure being performed,which may be input to processing unit 12 and/or machine learningalgorithm 11. As used herein, health care practitioners may include anyhealth care worker performing the steps indicated, such as physicians,nurses, technicians, operators, physician's assistants, and/or nursepractitioners, in various examples. In a specific example, sensor data22 may include a dictation that the patient's colon is spasming, whichmay be added to other input data received by processing unit 12, such asto annotate image input 14 with temporal information (sensor data 22)observed by the health care practitioner. Additionally or alternatively,sensor data 22 may include a verbal instruction or command from thehealth care practitioner to alter something within system 10, such as tochange a view or setting of imaging device 16. In a specific example,imaging device 16 may have two or more different recording modes, andsensor data 22 may include a sound input instruction to change from onetype of recording to another, such as to change imaging device 16 fromits standard recording mode to a near-infrared (NIR) mode, which may beused to better visualize a potential or suspected polyp or other medicalcondition state. In other examples, imaging device 16 may includecapabilities for white light endoscopy, autofluorescence imaging (e.g.,with magenta areas on the surface of potentially or suspected polyps),and/or magnifying endoscopy with narrow band imaging (e.g., to visualizea modified pit pattern of the mucosa with an increased number ofcapillaries), with the active mode being selected by sensor data 22(e.g., verbal instructions), in some examples. Patient health data 18and/or sensor data 22 (and/or dynamic state information 49 describedherein) may be input into processing unit 12 as an input tensor 24. Inother words, processing unit 12 may be configured to receive patienthealth data 18 and/or sensor data 22 as one or more input tensors 24.

As mentioned above, processing unit 12 may be programmed to perform anencoding conversion to encode patient health data 18 and thereby convertpatient health data 18 to encoded patient health data 20. In oneexample, patient health data 18 is converted to encoded patient healthdata 20 via one-hot encoding. For example, processing unit 12 and/ormachine learning algorithm 11 may include an encoding algorithm 26,which may also be referred to herein as an encoding system 26, that isconfigured to convert patient health data 18 into encoded patient healthdata 20. In some examples, encoding system 26 may encode patient healthdata 18 before it is received by processing unit 12. In some examples,encoding system 26 may be stored in one or more memories of processingunit 12. In a specific example, encoding system 26 may be configured toconvert data concerning patient risk factors into a numericalrepresentation based on a predefined data schema (e.g., a data tensor),thereby converting patient health data 18 to encoded patient health data20. Encoding algorithm 26 performs one or more processes that can berepeatedly executed on a given piece of information in the same way eachtime it is performed, via a set of highly defined rules, to produceencoded patient health data 20.

In some systems 10, processing unit 12 is further programmed to imageencode patient health data 18 in addition to (e.g., after) or instead ofperforming one-hot encoding conversion of patient health data 18.Additionally or alternatively, processing unit 12 and/or machinelearning algorithm 11 may be programmed to perform a reshaping operationon at least one image of image input 14 to change the dimensionality ofat least one image of image input 14 (e.g., transforming image input 14to a single column vector or to a multi-dimensional tensor). Forexample, said reshaping operation may include a flattening operation toflatten image input 14 to a tensor representation. In some examples,encoded patient health data 20 may be concatenated onto the tensorrepresentation of image input 14 (e.g., to the reshaped image input 14).Additionally or alternatively, processing unit 12 and/or machinelearning algorithm 11 may be programmed to perform a concatenatingoperation to concatenate encoded dynamic state information 49 onto atensor representation of least one image of image input 14. In otherwords, data from one or more sources (e.g., encoded patient health data20 and/or encoded dynamic state information 49) may be concatenated ontoreshaped image input 14 in some examples. As is understood in the art,flattening operations may be performed to reshape tensor data dimensionsinto a vector, which can then be appended to, or with, other data (e.g.,other one-dimensional data) within machine learning algorithm 11. Insome examples, the reshaping, flattening, and/or concatenating isperformed prior to or within a fully connected network portion ofmachine learning algorithm 11 (which may also be referred to herein as amulti-layer perceptron portion of machine learning algorithm 11).

As noted, system 10 is configured to perform one or more medicalcondition state determinations, via machine learning algorithm 11, basedon image input 14 and encoded patient health data 20. Such medicalcondition state determinations may be made continuously and/or in anautomated fashion after machine learning algorithm 11 has been trained.Additionally or alternatively, such medical state determinations may bemade in real-time, offering an improvement over existing prior artmachine learning algorithms. Specifically, machine learning algorithm 11may be configured to detect, classify, and/or localize one or moremedical condition states based on the one or more images from imagingdevice 16 (e.g., image input 14), patient health data 18, encodedpatient health data 20, and/or encoded dynamic state information 49. Inother words, as used herein, “medical condition state determinations”include detecting, classifying, and/or localizing medical conditionsshown in image input 14 (including determining a lack or absence of anymedical conditions), via machine learning algorithm 11. For example, inthe case of a colonoscopy procedure, machine learning algorithm 11 maydetect a polyp in one or more images of image input 14. Additionally,machine learning algorithm 11 may classify the polyp detected in animage. Such classification may be a simple binary classificationdifferentiating between the presence or absence of a polyp in a givenimage. In other examples, the classification may be more complex,selecting from among a plurality of classes of different types ofpolyps. Additionally or alternatively, machine learning algorithm 11 maylocalize the polyp, such as by pinpointing the location of region ofimage input 14 that contains the detected polyp, thereby determining thespecific location or region of the polyp within image input 14. Forexample, machine learning algorithm 11 may be configured to define theimage plane coordinates of the location of image input 14 in which apolyp was detected in the image frustum volume. In some examples, thisinformation may be used to guide future management and/orrecommendations pertaining to the detected medical condition state.

Such medical condition state determinations made by machine learningalgorithm 11 are displayed for and/or communicated to a user of system10, generally by converting image 30 of image input 14 to an outputimage 40, as schematically represented in FIG. 2. In some examples,encoding algorithm 26 of processing unit 12 produces output image 40,though in other examples, output image 40 may be produced directly bymachine learning algorithm 11. Generally, displaying the medicalcondition state determination(s) also includes displaying one or moreimages 30 from image input 14, and/or displaying encoded patient healthdata 20 along with the medical condition state determination, to createoutput image 40. In other words, system 10 is configured to providevisual output for (e.g., visualization of) medical condition statedeterminations via a graphical user interface (e.g., a display device42), with the visual output being augmented with patient health data 18such that information may be presented and viewed all together as asingle source (e.g., all together in output image 40).

This output image 40 can take many different forms, such as, forexample, using one or more bounding boxes 36 on an image, text 38, oneor more shaped outlines, one or more visual indications on a screen ormonitor, video, and/or one or more auditory signals (e.g., systems 10may be configured to beep or produce other sounds with increasing volumeand/or frequency depending on the level of the perceived risk currentlyshown). For example, as schematically represented in FIG. 2, one or morebounding boxes 36 may be displayed surrounding one or more features orportions of image 30, which may highlight or indicate the portions ofinterest of image 30 identified by machine learning algorithm 11. Forexample, if machine learning algorithm 11 detects a polyp in image 30,it may display bounding box 36 around said polyp in output image 40. Insome examples, such bounding boxes 36 may be displayed in cases wheremachine learning algorithm 11 has a threshold level of confidence withrespect to the feature within image 30. In other words, processing unit12 may be configured to determine a probabilistic diagnosis of themedical condition state of image input 14, based on image input 14 andencoded patient health data 20. Text 38 may include, for example, themedical condition state determination, a confidence level of theconclusion reached by machine learning algorithm 11, information as tolocalization and/or classification of the medical condition state,and/or other information about image 30 (e.g., patient information, timeor date at which image 30 was taken, etc.). To this end, system 10 mayinclude display device 42 (FIG. 1), such as a monitor or screen, that isconfigured to display, store, send, communicate, and/or print outputimage 40. In other words, output image 40 may include one or moreprinted output images and/or one or more digital output images. Machinelearning algorithm 11 may interface with display device 42 and/or otherhardware, communications systems, and/or apps to display and/orcommunicate information from the medical procedure and medical conditionstate determinations from machine learning algorithm 11. In someexamples, display device 42 includes a head mounted display, anaugmented reality device, a LCD (liquid crystal display) device, a LED(light emitting diode) device, and/or a plasma display device. Machinelearning algorithm 11 also may be configured to output recommendedtreatments and/or products as a result of the medical condition statedetermination.

In some examples, system 10 may be configured to initially display data(e.g., encoded patient health data 20) to a user or practitioner in anenlarged or more readable fashion, and then later compress the size ofthe displayed data such that it is less obtrusive. For example, whilemachine learning algorithm 11 may be able to be informed by a singlepixel of encoded patient health data 20, a human user may not be able tosee a single pixel, and/or may have an easier time understanding orinterpreting displayed data that includes labels, colored text, icons,diagram features, and/or larger areas of encoded data (e.g., collectionsof coded image pixels 34). In some examples, system 10 is configured todisplay real-time data or information during the medical imagingprocedure. In some examples, system 10 includes an image displayrendering function 41 (FIG. 1) which may be configured to render outputimage 40 to be displayed by display device 42. In some examples, imagedisplay rendering function 41 may be a component or feature of machinelearning algorithm 11, and/or image display rendering function 41 may bea component or feature of processing unit 12. In other examples, imagedisplay rendering function 41 may be a component or feature of displaydevice 42, or may be a standalone processing unit that creates outputimage 40 from the results of machine learning algorithm 11. Imagedisplay rendering function 41 may be configured to overlay, append, add,integrate, overwrite, and/or otherwise display encoded patient healthdata 20 and/or encoded dynamic state information 49 onto one or moreimages of image input 14, to be displayed and/or communicated by displaydevice 42 as output image 40.

Specific examples of data that may be displayed to a user orpractitioner during the medical procedure and/or in output image 40 mayinclude polyp count (with presumed pathology displayed and appropriatelymodifying the previously established pretest probabilities), predicteddistance into the colon, upcoming landmarks and displaying currentlandmarks (such as ileocecal valve, appendiceal orifice, spleen, liver,terminal ileum, etc.), information from previous colonoscopies aboutpolypectomies (and method used), tattooed colons, previous surgicalresections, diverticula locations (which may assist in reducing theprobability that an outpouching is not a polyp but instead an invertedcolonic diverticulum), recommendations on anesthesia, alerts regardinganesthesia (e.g., if the patient is awakening, machine learningalgorithm 11 could display a suggestion for additional anesthesia, ahigher concentration of anesthesia dosage, and/or further titration ofanesthetic medications), alerts requiring assistance (e.g. if thepatient begins to cough, lidocaine could be suggested by machinelearning algorithm 11), information regarding when imaging device 16 isapproaching areas that were not well visualized on a previouscolonoscopy, predictions as to the current part of the colon imagingdevice 16 is located in (as location heavily influences polyp andcolonic cancer probability rates), a live probability of finding apolyp, a live probability of pathology of the polyp, the last givenmedication(s), suggestions for new medications to be administered (e.g.propofol, lidocaine, etc.), information on renal/hepatic clearance,volume distribution, liver failure, and/or creatinine level.Additionally or alternatively, systems 10 may display data to the useror practitioner at the end of the medical imaging procedure, which mayinclude information such as the predicted/recommended return year for asubsequent medical imaging procedure, information about the currentmedical imaging procedure to assist billing, documentation, andself-improvement (e.g., time and method spent removing polyps, time tocecal intubation, time withdrawing, etc.), a predicted model of thecolon with areas poorly visualized on it to assist any subsequentcolonoscopy to ensure complete observation of colon, a procedureduration and event timestamps based on visually identified patientanatomical features, and/or a summary of the colonoscopy procedure withrelevant details and screen captures of polyps. Such information may, insome examples, be provided in a billable report that automaticallygenerated for the patient by disclosed systems 10. Of course similarmetrics and relevant data may be provided and displayed as describedherein for medical imaging procedures other than colonoscopies as well.

Additionally or alternatively, systems 10 may be configured to produce asound (e.g., an auditory signal) to communicate information aboutfindings from machine learning algorithm 11, one or more medicalcondition state determinations, or etc. In some examples of system 10,processing unit 12 is programmed to cause display device 42 to emit suchsounds, and/or processing unit 12 itself may be configured to producethe auditory signals, such as via a speaker or other auditory outputdevice. For example, system 10 may be configured to produce a particularsound when a particular medical condition is detected, such as a beep orother sound when a polyp is detected, in real-time during the medicalprocedure. In some examples, the volume (e.g., magnitude and/oramplitude), frequency, and/or tone or type of sound may change dependingon (e.g., proportionally to) the type of medical condition detected, theclassification or seriousness of the medical condition detected, and/orthe confidence of machine learning algorithm 11 in the determination.For example, the produced sound may be louder when machine learningalgorithm 11 reports a higher confidence level in a medical conditionstate determination, and quieter when machine learning algorithm 11reports a lower confidence level in the medical condition statedetermination. In some examples, the auditory signal may be apre-recorded sound or synthesized voice announcement of a medicalcondition state determined by machine learning algorithm 11 and/or theassociated confidence level of said medical condition statedetermination. Additionally or alternatively, the pre-recorded sound orsynthesized voice may include information about the confidence level ofmachine learning algorithm 11 and/or recommended corrective action. Insome examples, the pre-recorded sound or synthesized voice may includeinformation warnings or alerts, such as maintenance actions recommendedfor the system. In an illustrative example, one auditory signal mayprovide information such as “polyp detected with 52% confidence,lighting is poor, clean camera,” which can alert a user to a need forbetter preparing the system and/or environment for more accurate inputinformation.

Additionally or alternatively, systems 10 may be configured toautomatically generate an output report, or summary report, for thehealth care practitioner and/or for the patient. For example, systems 10may be configured to generate this report at the end of procedure, whichmay provide the patient a summary (e.g., printed and/or in an electronicformat) of the findings (e.g., the medical condition statedetermination), future appointments or procedures that are scheduled,and/or treatment recommendations. Additionally or alternatively, suchoutput reports may include one or more output images 40 produced bysystems 10. Additionally or alternatively, output reports generated bysystems 10 may include billing information for the procedure. In someexamples, systems 10 may be configured to ask or prompt the health carepractitioner for a confirmation of the procedure (e.g., “An adenomatouspolyp was piecemeal removed and initially 8×5 cm in the sigmoid colon;it was removed with hot biopsy forceps; is that correct?”), to which thepractitioner may be able to respond verbally and/or via an input buttonor key. Systems 10 may be configured to ask or prompt follow-upquestions, such as “Would you recommend the patient to be re-examined in3 months?” Responses from the practitioner and/or the automaticallygenerated output reports including the same may be recorded by systems10 for storage in the patient's medical records (e.g., systems 10 may beconfigured to electronically transfer the output report to the patient'selectronic health record), for output reports generated by systems 10,and/or for reporting to the patient's insurance company. In someexamples, systems 10 may be configured to automatically assign a billingcode (e.g., from a lookup table) for the procedure performed. In someexamples, the output report generated by systems 10 may include imagesof the polyp (or other medical condition) before and after removal,expected distance from rectum or other anatomical features, and/or a mapor image of the 3D expected polyp location within the patient's body.For example, the output report may mark or indicate the type of polypfound, what procedure was performed, and/or recommended follow-up (e.g.,return in 3 months for a repeat or other monitoring). Output reportsgenerated by systems 10 also may indicate areas where visualization wasless than ideal. In some examples, systems 10 may be configured togenerate different output reports for different parties, such as oneoutput report for the patient with information relevant to the patient,and a different output report for the patient's insurance company, whichmay include information specifically needed by the insurance company butless useful to the patient.

Turning now to FIGS. 3-8, illustrative non-exclusive examples of outputimages 40 are illustrated. Where appropriate, the reference numeralsfrom the schematic illustrations of FIGS. 1-2 are used to designatecorresponding parts in FIGS. 3-8; however, the examples of FIGS. 3-8 arenon-exclusive and do not limit output images 40 to the illustratedexamples. That is, output images 40 are not limited to the specificexamples illustrated in FIGS. 3-8, and may incorporate any number of thevarious aspects, configurations, characteristics, properties, etc. thatare illustrated in and discussed with reference to the schematicrepresentations of FIGS. 1-2 and/or the examples of FIGS. 3-8, as wellas variations thereof, without requiring the inclusion of all suchaspects, configurations, characteristics, properties, etc. For thepurpose of brevity, each previously discussed component, part, portion,aspect, region, etc. or variants thereof may not be discussed,illustrated, and/or labeled again with respect to each of FIGS. 3-8;however, it is within the scope of the present disclosure that thepreviously discussed features, variants, etc. may be utilized therewith.

FIGS. 3-6 show illustrative examples of output images 40 that may beproduced by presently disclosed systems 10. In FIG. 3, output image 40shows an image from a colonoscopy procedure (e.g., an image input froman imaging device according to one example of system 10), with text 38indicating a medical condition state determination, along with theprobabilistic determination of that medical condition statedetermination (e.g., the confidence of machine learning algorithm 11 inthe medical condition state determination). In the example of FIG. 3,text 38 indicates that machine learning algorithm 11 determined thatthere is an 83% probability with a confidence interval that output image40 includes a polyp, based on the training model. In the example of FIG.4, output image 40 includes the same image from the colonoscopyprocedure, and text 38 indicates an 83% probability that output image 40shows a hyperplastic polyp, based on the training model. In other words,in the example of FIG. 4, output image 40 includes the image along withthe expected pathology type shown. In this example, text 38 indicates aclassification of the feature (e.g., the polyp) detected by machinelearning algorithm 11.

In the example of FIG. 5, output image 40 includes the same text 38 asin FIG. 4 (with the classification and probability of the medicalcondition state determination), and also includes bounding box 36 thathas been added to the image input such that bounding box surrounds, oris positioned around, the feature in question. While bounding box 36 isillustrated as rectangular in shape, bounding box 36 may be any desiredshape, such as circular, polygonal, a line or combination of lines, anarrow, a shaped outline roughly matching the identified medicalcondition, and/or any other graphical indication as to the location ofthe detected medical condition on the image. Specifically, bounding box36 is positioned on the image input to indicate the area within theimage where machine learning algorithm 11 detected a polyp (or otherfeature, for other examples of system 10). Bounding box 36 may bedisplayed in a color that is optimized to highlight the presence ofbounding box 36 (e.g., bounding box 36 may be displayed in a color thathas a high contrast compared to the background image on which it isoverlaid). Additionally or alternatively, bounding box 36 may bedisplayed in a particular color according to a coding scheme. Forexample, different colors of bounding boxes 36 may be used to indicate adifferent type or severity of medical condition state, and/or differentcolors of bounding boxes 36 may be used to indicate different confidencelevels of machine learning algorithm 11. Additionally or alternatively,the actual feature of interest (i.e., a polyp) may have its appearanceand/or color altered by machine learning algorithm 11 to highlight thefeature in output image 40.

In the example of FIG. 5, output image 40 includes the image input,pathology information, and bounding box 36, along with the medicalcondition state determination indicated by text 38. In the example ofFIG. 6, output image 40 includes the same image input, text 38, andbounding box 36, and also includes risk factors 50, which may berepresented in output image 40 using additional text and/or images oricons. Risk factors 50 may include those risk factors present in thegiven patient that inform the medical condition state determination madeby machine learning algorithm 11. For example, if smoking is a knownrisk factor for a particular type of polyp and the image inputs beinganalyzed by machine learning algorithm 11 are from a patient who smokes,output image 40 may include an indication in risk factors 50 that thepatient is a smoker. Additionally or alternatively, output image 40 mayinclude semantic segmentation to focus machine learning algorithm 11 ona specific area of image input 14 and/or simplify image input 14 and/oroutput image 40.

Output image 40 may include additional or alternative information thanis shown in FIGS. 3-6, such as current polyp count in real-time duringthe medical imaging procedure (or a current real-time count of anothermedical condition in types of procedures other than colonoscopies),predicted distance of travel of the imaging device within the patient'sbody, upcoming anatomical or physical landmarks or markers within thepatient's body, information from previously performed medicalprocedures, recommendations on anesthesia, probability rates of cancerin a given area of the patient's body, a live probability of finding apolyp or other medical condition, a live probability of the pathology ofa polyp or other detected medical condition, information on the mostrecent medication(s) the patient received, a predicated date forsubsequent procedures, a predicted model of an organ of the patientbeing imaged, and/or summary information regarding the medical imagingprocedure. Of course when the medical imaging procedure being evaluatedis one other than a colonoscopy, such alternative information may beprovided that specifically pertains to measurements, indications, and/orrecommendations associated with that particular imaging procedure. Forexample, rather than polyp count, a machine learning algorithm 11configured to evaluate images from an endoscopy of a patient's esophagusmay display information about the number of tumors found during thatendoscopy.

In some systems 10, processing unit 12 is configured to add encodedpatient health data 20 to at least one image of image input 14, whichmay include appending encoded patient health data 20 to image trainingdata (e.g., image input 14), overlaying encoded patient health data 20onto the image training data, embedding encoded patient health data 20into the image training data, and/or otherwise adding encoded patienthealth data 20 to the image training data such that machine learningalgorithm 11 is informed and trained using both the image training dataand encoded patient health data 20. In some examples, encoded patienthealth data 20 is embedded into at least one image of image input 14 ator before a time that machine learning algorithm 11 analyzes image input14, such that machine learning algorithm 11 analyzes image input 14together with encoded patient health data 20 embedded in the at leastone image of image input 14.

Systems 10 may embed and display encoded patient health data 20 within aparticular, predetermined region of at least one image of image input 14for analysis by machine learning algorithm 11 and/or to display to oneor more users of system 10. For example, as shown in FIG. 2, image input14 may include a plurality of images 30, which may be taken from videofootage, or may be photographs or other medical imaging output images.One or more images 30 may be converted by processing unit 12 such thatencoded patient health data 20 is added to the one or more images 30. Inthe example of FIG. 2, encoded patient health data 20 is shown embeddedin one of images 30 by processing unit 12, with encoded patient healthdata 20 being positioned within a region 32 of image 30. In FIG. 2,region 32 is shown as being in the lower left hand corner of image 30,though in other examples, region 32 may be in any desired location withrespect to image 30. In other various examples, region 32 may be locatedin the lower right corner, the upper right corner, the upper leftcorner, along the left side, along the right side, along the top, alongthe bottom, in a middle area, and/or within a perimeter of image 30.Region 32 generally is the same for each image 30 into which encodedpatient health data 20 is embedded. For example, system 10 may beconfigured to display encoded patient health data 20 within the sameregion 32 each time images 30 are analyzed, so that users may easilyfind said encoded patient health data 20 within image 30. Some encodedpatient health data 20 may be displayed for a temporary period of timeduring the procedure and/or analysis, and/or may be updated throughoutthe procedure. Some encoded patient health data 20 may be displayed forthe duration of the procedure and/or analysis.

Machine learning algorithm 11 may be configured to process and interpretencoded patient health data 20 that is embedded in at least one image 30in this manner. For example, encoded patient health data 20 may beencoded and embedded into image 30 as collections of coded image pixels34 that are added to the image input (e.g., by image display renderingfunction 41). Collections of coded image pixels 34 may be any desiredshape, size, and/or arrangement. In some examples, collections of codedimage pixels 34 may include QR codes®, microQR codes, IQR codes,QRGraphy, Frame QR, HCC2D, microQRJAB codes, JAB codes, ArUco codes,barcodes, one or more pixels configured to be detectable by machinelearning algorithm 11, and/or one or more pixels arranged to be visibleto a human eye. In other words, collections of coded image pixels 34 mayemploy any of various known data compression or encoding techniques.

For example, FIG. 7 illustrates a specific example of encoded patienthealth data 20, shown as a plurality of collections of coded imagepixels 34 (e.g., 34 a, 34 b, 34 c, 34 d, and 34 e) added to region 32 ofimage 30. Again, said encoded patient health data 20 is added to image30 by processing unit 12 of presently disclosed systems 10, such as byadding collections of coded image pixels 34 to image 30, though encodedpatient health data 20 may be added to image 30 in other forms, inaddition to or instead of said collections of coded image pixels 34. Asthe term is used herein, “adding” encoded patient health data 20 toimage 30 includes appending collections of coded pixels 34 (or otherforms of encoded patient health data 20) to image 30, and/or overlayingcollections of coded image pixels 34 onto image 30. In other words, theterm “adding” includes overwriting, appending, adding, padding,embedding, and/or other means of incorporating collections of codedimage pixels 34 (or other representation of encoded patient health data20) into image input 14. In some examples, collections of coded imagepixels 34 may be overlaid onto image 30 by overwriting a portion of thepixels forming image 30, which may be performed when it is desired toretain the original dimensions of image 30. In other examples,collections of coded image pixels 34 may be appended to image 30, suchas by being appended along one or more edges of image 30 (e.g., alongthe bottom of image 30), which may thereby change (e.g., increase) theoverall dimensions of image 30. Encoded patient health data 20 (e.g.,collections of coded image pixels 34) are generally displayed on outputimages 40 as well, after analysis by machine learning algorithm 11.

Collections of coded image pixels 34 may be configured such that theyare discernible and understood by a human observer of system 10, as wellas configured to by analyzed by and deliver information to machinelearning algorithm 11. In some examples, said collections of coded imagepixels 34 may encode patient health data 18 (FIG. 1) using a color-codedscheme. For example, each different type of patient health data 18 maybe encoded using a different color, with different values beingrepresented by the shade or darkness of that color (e.g., a respectiveshade of each respective collection of coded image pixels 34 mayrepresent the relative value of the respective encoded piece of patienthealth data encoded in the respective collection of coded image pixels34). In this manner, a plurality of collections of coded image pixels 34may be generated such that a respective collection of coded image pixels34 is displayed for each respective type or category of encoded patienthealth data 20.

As an illustrative example, collection of coded image pixels 34 a may bea blue collection of pixels representing the patient's age, with thepatient's age represented by the shade of blue shown (e.g., older agesmay be shown in darker blue, while younger ages may be shown in a lightblue). As other illustrative examples, collection of coded image pixels34 b may represent, or encode, the patient's gender using red pixels,collection of coded image pixels 34 c may encode the patient's raceusing green pixels, collection of coded image pixels 34 d may encode thepatient's smoking habits using purple pixels, and collection of codedimage pixels 34 e may encode the patient's drug use using black pixels.Of course, these examples are only illustrative. Encoded patient healthdata 20 may be embedded into image 30 showing more or fewer differentmetrics or types of patient data, using textures or patterns, usingdifferent colors, and/or arranged differently than shown in thisillustrative example. In some examples, collections of coded imagepixels 34 may be encoded in gray-scale rather than color. In someexamples, collections of coded image pixels 34 may be encoded usingdifferent colors for a single type of patient data (e.g., patient agemay be represented by different colors of pixels for different ageranges, such as orange for one age range, and red for another agerange). Collections of coded image pixels 34 also may be emphasized orhighlighted in some examples, such as to flag for a practitioner aparticularly relevant piece of patient health data.

In some examples, such as shown in FIG. 8, an icon 44 may be displayed(e.g., in output image 40) to aid or facilitate visualization andinterpretation of output image 40 by a human user (e.g., a physician ora technician using system 10). In other words, system 10 may beconfigured to display a respective icon 44 for each respectivecollection of coded image pixels 34 to indicate what the respectivecollection of coded image pixels 34 is encoding. For example, FIG. 8shows a cigarette-shaped icon 44 positioned adjacent collection of codedimage pixels 34 d, which may be configured to encode the patient'ssmoking history. Said icons 44 may serve as a visual cue to human usersas to what each collection of coded image pixels 34 represents orencodes, rather than requiring human users to remember what eachrespective color (or pattern, or texture, etc.) refers to. Additionallyor alternatively, text labels may be displayed on output image 40 as areminder of the meaning of collections of coded image pixels 34. Icons44 and/or other types of labels may be produced by machine learningalgorithm 11 and/or processing unit 12 (e.g., encoding algorithm 26 ofprocessing unit 12), to be displayed in output images 40 produced bymachine learning algorithm 11 after analyzing image input 14.Visualization aids such as icons 44 and/or labels may be displayed inreal-time during the medical procedure, for real-time visualization andaggregation of information for the practitioner/user of systems 10.

Collections of coded image pixels 34 may be arranged in any suitablefashion when added to images 30. For example, collections of coded imagepixels 34 may be arranged in a row, a column, and/or an array on one ormore images 30. Generally, collections of coded image pixels 34 arepositioned together, such that they are relatively close together, oradjacent each other. For example, collections of coded image pixels 34may be positioned together such that they are all positioned within agiven region 32 of image 30. In some examples, however, collections ofcoded image pixels 34 may be spread out or distributed in differentregions 32 of image 30. For example, one or more collections of codedimage pixels 34 may be located or displayed in one region 32, while oneor more other collections of coded image pixels 34 may be located ordisplayed in a different region 32 of image 30.

Some systems 10 may include computing device 28 configured forcollecting and/or retrieving patient health data 18, with computingdevice 28 being further configured to deliver patient health data 18 toprocessing unit 12. Additionally or alternatively, computing device 28may be configured to collect and/or retrieve patient health data 18 inreal-time from a database, such as from electronic health records and/orelectronic medical records. In some systems 10, processing unit 12itself may perform this function, while in other systems 10, processingunit 12 may access this information from computing device 28 or fromanother source. Patient health data 18 may include static data and/orprocessed information.

Some systems 10 may include an apparatus 46 for determining dynamicstate information 48 of a patient, with said apparatus 46 also beingreferred to as a dynamic state apparatus 46. Dynamic state apparatus 46may be configured to deliver dynamic state information 48 to processingunit 12. Dynamic state information 48 is information about the patienttaken contemporaneously with a medical procedure that produces imageinput 14, which may be continuously changing in real-time. For example,dynamic state information 48 may be sensor-derived data obtained inreal-time during a medical imaging procedure that produces image input14. Dynamic state information 48 may be used to aid interpretation ofimages 30 obtained from imaging device 16 (e.g., dynamic stateinformation 48 may provide information about the contents of images 30that would not be obtainable by simply observing images 30 themselves.For example, increases in a patient's blood pressure may cause flaringof polyps present, which may change their appearance, such as makingthem appear larger than normal, and etc. Thus, when analyzing a givenimage 30 of image input 14, the patient's blood pressure (and/or otherdynamic, real-time information at the time of the image) may be used inmaking medical condition state determinations with respect to thatimage. Similarly, if the patient's tissues are bleeding at all duringthe medical procedure, this can alter the appearance of polyps and othermedical conditions within the patient, which machine learning algorithm11 may be trained to identify and interpret. In other examples, dynamicstate information 48 can dynamically affect the visual appearance ofcancers or other medical conditions, and thus recording and storing thisdynamic state information 48 may facilitate the medical condition statedeterminations made by machine learning algorithm 11.

Dynamic state information 48 may include, for example, the patient'sheart rate during the procedure (i.e., real-time heart rateinformation), the patient's blood pressure during the procedure (i.e.,real-time blood pressure information), compensated heart rate,anesthetics used during the procedure, telemetry, saline or other fluidsused during the procedure, the patient's oxygen saturation during theprocedure, end tidal carbon dioxide (capnography), the patient's currentmedications (e.g., those taken the day of the procedure), activity indistal extremities, positional data pertaining to imaging device 16relative to the patient's body, positional data pertaining to sensorsbeing used for measurements, temperature information (internal orexternal), and/or information regarding previous injuries or proceduresperformed on the patient such that image inputs 14 may be correspondedor mapped to areas within the patient's body that were previouslyinjured, where applicable. Additionally or alternatively, dynamic stateinformation 48 may include information regarding a patient's cardiaccycle and/or breathing cycle, such that image input 14 may be temporallyannotated such that each of the one or more images 30 from imagingdevice 16 may be matched with a phase of the patient's cardiac cycleand/or breathing cycle. For example, with respect to the patient'scardiac cycle, one or more images from image input 14 each may belabeled as corresponding to diastole or systole, based on mappingrespective time stamps from the respective image to time stamps fromcardiac cycle data taken during the procedure. Additionally oralternatively, one or more images from image input 14 each may belabeled corresponding to P-wave, QRS complex, T-wave, and so on, of thepatient's cardiac cycle. In various examples of system 10, dynamic stateapparatus 46 may include one or more motion sensors, one or moreaccelerometers, and/or one or more video cameras configured for motiondetection.

Similar to patient health data 18, dynamic state information 48 may beencoded by processing unit 12 to form encoded dynamic state information49. Encoded dynamic state information 49 may be embedded into at leastone image of image input 14, such as at or before a time that machinelearning algorithm 11 analyzes image input 14. In this manner, machinelearning algorithm 11 may be configured to analyze image input 14together with dynamic state information 48 embedded in the at least oneimage of image input 14. Processing unit 12 is programmed to performone-hot encoding conversion to encode dynamic state information 48, insome examples, such as via encoding system 26, which may also bereferred to herein as an encoding algorithm 26. Processing unit 12 alsomay be configured to perform image encoding on dynamic state information48, which may be performed after the one-hot encoding.

Machine learning algorithm 11 may be trained using manual, orsupervised, labeling of image inputs 14 used during training.Additionally or alternatively, machine learning algorithm 11 may betrained using semi-supervised labeling information of image inputs 14used during training. In some examples, the machine learning algorithm11 may be configured to first receive supervised labeling, and then toreceive semi-supervised labeling, and then to perform auto-labeling. Insome examples, machine learning algorithm 11 is configured to firstreceive supervised labeling and/or to perform auto-labeling, and then toreceive semi-supervised labeling to refine and improve labels in adataset. In other words, system 10 may be configured to perform alabeling feedback loop that includes semi-supervised labeling, with suchlabeling feedback loop functioning to improve training efficiency ofmachine learning algorithm 11. In other words, manual labeling orsemi-supervised labeling may be performed to assign a plurality ofinitial labels, followed by manual verification for at least a portionof the initial labels, to complete said labeling feedback loop.

This labeling feedback loop may enable machine learning algorithm 11 tobe trained using “big data,” which was not used for prior art machinelearning models because they were not designed to continuously collectand aggregate large amounts of data. Prior art academic studiestypically were designed to operate with a discrete and fixed sizedataset. As opposed to prior art machine learning models, presentlydisclosed machine learning algorithms 11 may be configured to becontinuously operating systems to automatically collect, aggregate, andorganize data to streamline transmitting new data into machine learningalgorithm 11. Thus, system 10 may be configured to receive big data fortraining machine learning algorithm 11, which can continuously improveperformance of machine learning algorithm 11 as compared to prior artmachine learning models, such as by improving accuracy and/or improveefficiency in learning patterns within the data. As used herein, bigdata currently includes datasets that are 1 terabyte or larger, thoughthis definition is fluid and understandings may change over time as towhat constitutes big data in the future. Additionally, because disclosedmachine learning algorithms 11 are configured to automatically process,aggregate, and anonymize (e.g., de-identify) data (e.g., in an automatedmanner), this enables greater access to patient health data by making iteasier or less onerous to comply with regulations protecting such data(e.g., because the automated nature of machine learning algorithms 11may avoid the need for humans to directly access the protected data). Insome examples, system 10 is configured for online learning such thatmachine learning algorithm 11 is updated using neural network weights,and thereby continues to learn as it receives additional image input 14and patient health data 18 during use (e.g., after the initial trainingphase).

Thus, disclosed systems 10 may be configured to offer many advantagesand features over prior art machine learning models. Presently disclosedsystems 10 and machine learning algorithms 11 are repeatable, such thatthe overall machine learning algorithm 11 (including de-identificationof patient health data and labeling processes) may be repeated anynumber of times for training and/or deployment of disclosed machinelearning algorithms 11. The scale of data that may be used to train, andmay be used as input to, presently disclosed machine learning algorithms11 may be larger than possible with prior art machine learning models.Presently disclosed machine learning algorithms 11 may be configured toperform medical condition state determinations in an automated,continuous, and/or real-time fashion. Labeling feedback loops withsemi-supervised labeling that are utilized by presently disclosedmachine learning algorithms 11 can provide improved algorithm trainingefficiency and access to large scale data and/or big data, as comparedto prior art machine learning models.

Furthermore, presently disclosed systems 10 encode patient health data(e.g., with one-hot encoding and/or reshaping operations) such thatmachine learning algorithms 11 make use of this meta-data in addition toimage data, whereas prior art machine learning models make use only ofimage data. Thus, neural networks of the presently disclosed machinelearning algorithms 11 are informed by both images from medicalprocedures and by encoded patient health data (generally originating astext data), which enables disclosed systems 10 to provide visual outputshowing the medical condition state determination that is augmented withconsideration to patient history and other relevant patient health data,which is another improvement over prior art machine learning models.Thereby, systems 10 advantageously may be configured to informphysicians or other practitioners of patient history from a singlesource (e.g., output image 40), which may help enable the physician toidentify patterns that are harder to observe when the data sources areseparated, such as is the case in prior art imaging systems. Similarly,machine learning algorithm 11 may be able to observe patterns betweenpatient history information and the medical procedure images that werenot previously considered by prior art models or understandings, by theability provided by systems 10 for machine learning algorithm 11 tosimultaneously consider patient history along with the images from themedical procedure. These features may enable disclosed systems 10 toproduce improved accuracy in its medical condition state determinations.

While FIGS. 1-8 give a high level representation of presently disclosedsystems 10, FIGS. 9-15 schematically provide flowcharts that representillustrative, non-exclusive examples of methods according to the presentdisclosure. In FIGS. 9-15, some steps are illustrated in dashed boxesindicating that such steps may be optional or may correspond to anoptional version of a method according to the present disclosure. Thatsaid, not all methods according to the present disclosure are requiredto include the steps illustrated in solid boxes. The methods and stepsillustrated in FIGS. 9-15 are not limiting and other methods and stepsare within the scope of the present disclosure, including methods havinggreater than or fewer than the number of steps illustrated, asunderstood from the discussions herein.

FIG. 9 gives a high level flowchart representation of related methods100 associated with said systems 10. Overall, methods 100 generallyinclude acquiring or collecting data to train machine learning algorithm11, at 102, preparing the data for training machine learning algorithm11, at 104, training machine learning algorithm 11, at 106, anddeploying machine learning algorithm 11, at 108. In some examples,different method steps may be performed by different parties. Forexample, researchers or developers may primarily perform data collectionat 102, data preparation at 104, and training at 106, whilepractitioners or other researchers may perform deployment at 108. FIG. 9also illustrates additional aspects of each of these overall steps, eachof which will be described in further detail herein.

Acquiring and/or collecting data at 102 may include, for example,procuring medical reports or datasets at 110. Additionally oralternatively, acquiring and/or collecting data at 102 may includecollecting or acquiring data from electronic health records, electronicmedical records, other health/medical records or charts, and/or from thepatient directly, and may include acquiring and/or collecting data frommultiple sources of any of the types of patient health data 18 discussedabove. Preparing the data at 104 may include data extraction at 112and/or data de-identification at 114. For example, data extraction at112 may include extracting images from the data (e.g., medical reports)acquired at 110, and extracting text from the acquired data. In someexamples, data extraction at 112 includes converting data extracted fromthe acquired data into a CSV file for later use by machine learningalgorithm 11. Acquired datasets often include patient-identifyinginformation, and this information may be removed during datade-identification at 114. Generally, de-identification of the data at114 includes eliminating any identification of the patient associatedwith the meta-data (e.g., patient health data 18) in a database. Thedatabase may be a text-based database, though other database formatsalso may be used. For example, said database may contain the datasources in various forms: text, numerical, date, data array, and/or etc.In some examples, the data de-identification is automated. In otherwords, a computer system (e.g., processing unit 12, machine learningalgorithm 11, and/or another computing system) may automaticallyde-identify the acquired data, at 114. In various examples, datade-identification at 114 may include assigning a universally uniqueidentifier (UUID) to each individual represented in the acquireddataset(s), elimination of features from the acquired dataset(s),computing the patient's age, generation of a master database containingthe original acquired dataset(s), and/or generation of a de-identifieddatabase containing the de-identified information to be used in trainingmachine learning algorithm 11.

Once the datasets are acquired at 102, the data may be further preparedat 104, via manual labeling at 116 and/or semi-supervised labeling at118, and then used to train machine learning algorithm 11 at 106, suchas via artificial intelligence training at 120. In some examples,preparing the data at 102 to train machine learning algorithm 11 at 106includes both manual labeling at 116 (which also may be referred to assupervised learning) and semi-supervised labeling at 118, withsemi-supervised labeling at 118 being performed after manual labeling at116. In some examples, preparing the data at 104 may includeauto-labeling of the biological images, performed by machine learningalgorithm 11.

Artificial intelligence training at 106, 120 may be phased, withtraining increasing in complexity as performance of machine learningalgorithm 11 improves during training. For example, artificialintelligence training at 120 may include at least one phase, at leasttwo phases, at least three phases, at least four phases, at least fivephases, and/or six or more different phases. In a specific example, afirst phase of artificial intelligence training at 120 may includeevaluating model effectiveness based on visual interpretations of eachrespective biological image from the current model, a second phase mayinclude measuring prediction accuracy based on respective pathologyresults for the respective biological specimen shown in the respectivein situ biological image, a third phase may include updating the modelweights based on the measured prediction accuracy of the respectivepathology results for the respective biological specimen shown in therespective in situ biological image, in combination with bounding boxes36 to identify an area of interest within the respective in situbiological image, and a fourth phase may include measuring accuracybased on respective pathology results for the respective biologicalspecimen shown in the respective in situ biological image, incombination with bounding boxes 36 to identify an area of interestwithin the respective in situ biological image, and further based on thepatient health data for the respective patient. Said phases ofartificial intelligence training at 120 may be implemented serially orin parallel to train machine learning algorithm 11. In some examples, arespective dataset may be obtained for each respective phase of theplurality of training phases at 120, such that methods 100 may utilize adifferent respective dataset for each respective phase of the pluralityof training phases at 120.

Once machine learning algorithm 11 is trained at 106, deployment at 108may be performed. In other words, training machine learning algorithm 11at 106 to classify, detect, and/or localize one or more medicalcondition states can enable said machine learning algorithm 11 to bedeployed at 108 to classify, detect, and/or localize one or more medicalcondition states in new biological images obtained from new patients(e.g., biological images from patients that were not used in trainingmachine learning algorithm 11).

FIGS. 10-15 break down individual steps of methods 100 from FIG. 9 intofurther details. FIG. 10 illustrates a more detailed view of acquiringor collecting data, at 102. Acquiring or collecting data at 102generally begins by performing a medical procedure at 124. Said medicalprocedure may be prospective or retrospective in various examples.Performing the medical procedure at 124 may include performing anymedical procedure that produces biological images and/or biologicalspecimens. For example, performing the medical procedure at 124 mayinclude performing a sonography procedure, an x-ray, a CT scan, an MRI,a PET scan, retinal imaging, dermatoscopy, radiography, a mammogram,endoscopy (e.g., lower endoscopy or upper endoscopy), a colonoscopy, anesophagogastroduodenoscopy, a bronchoscopy, photoacoustic endoscopy, aprocedure using an electro-optical sensor, NBI colonoscopy, white lightendoscopy, chromoendoscopy, and/or a procedure using a capsule endoscopydevice. Additionally or alternatively, performing the medical procedureat 124 may include performing surgery, an excision, and/or a biopsy.Additionally or alternatively, performing the medical procedure at 124may include examining the patient's external body, cancer detection,assessment of auto-immune diseases, assessment of drug-induced injuries,assessment of trauma-induced injuries, and/or infection assessment.Performing the medical procedure at 124 results in, or enables,procurement of one or more biological specimens at 126, obtaining one ormore in situ biological images at 128, and/or acquisition of prospectivepatient health data at 130, which is a subset of procuring the medicalreports at 110 of FIG. 9. Acquisition of prospective patient health dataat 130 may include obtaining a plurality of medical reports for trainingmachine learning algorithm 11, which may be obtained from, for example,electronic health records and/or electronic medical records.

In examples where acquiring or collecting the dataset at 102 includesprocuring biological specimens at 126, said specimens may undergohistopathological analysis at 132, which may be used to generate apathology report at 134. In other words, acquiring or collecting data at102 may incorporate acquiring pathology results for the one or morebiological specimens procured at 126. The specimen's pathology resultsmay then be extracted, at 136, for inclusion in the dataset. In situbiological images of the specimen may be obtained at 128 in parallelwith procurement of the biological specimen at 126, instead ofprocurement of biological specimens at 126, or before or afterprocurement of biological specimens at 126. In some examples, eachbiological image of the at least one in situ biological image obtainedat 128 is at least 578×462 pixels, at least 1280×720 pixels, and/or atleast 720×486 pixels in size. Prior art training methods used to trainprior art machine learning models often could not or did not use theselarger images as input, whereas disclosed machine learning algorithms 11may be configured to accept such sizes of images as input. For example,publicly available models for polyp detection use lower resolutionimages as input, such as YOLO v3, which uses an image input of 416×416pixels, and resnet152, which uses an image input of 224×224 pixels. Notusing larger images as input limits the accuracy of some prior artmachine learning models in interpreting the images and thus in makingmedical condition state determinations.

Said biological images may be annotated by a health care practitioner,such as a physician, a nurse, nurse practitioner, physician's assistant,or a technician, at 138. For example, a physician may annotate abiological image at 138 by indicating what is shown in the image,classifying a medical condition shown in the image, indicating where inthe biological image a medical condition is visible, and/or measuringand indicating a size or dimensions of a medical condition in thebiological image. As another example, annotation at 138 may includeannotation by the patient themselves, such as in cases where machinelearning algorithm 11 and/or processing unit 12 may interact with thepatient to enter demographic or other information. Additionally oralternatively, a post-procedure report may be annotated at 140, and areport (e.g., a Provation® report) may be generated with respect to thebiological image, at 142. Also, a patient health data report may begenerated at 144 with the patient health data acquired at 130. Thus, thecombination of the pathology results extracted at 136, the reportgenerated about the biological images at 142, and the patient healthdata report generated at 144 for a plurality of patients and theirrespective procedures/biological specimens may together form a datasetthat may be used to train machine learning algorithm 11. In someexamples, de-identification may be performed at 114 on the pathologyresults extracted at 136 and/or on the report generated about thebiological images at 142, and/or said data may be linked at 143, priorto generating the patient health data report at 144.

Said dataset acquired and/or collected at 102 may be manually labeled at116 and/or subject to semi-supervised labeling at 118. With reference toFIG. 11, the specimen pathology results extracted at 136 may be used tocreate an association between the images and the pathology results at146. Such association may be created automatically at 146, such as bymachine learning algorithm 11. Additionally or alternatively, apractitioner may create associations between the images and thepathology results at 148. These associations created at 146 and/or 148may be used to label the images from the medical procedure, at 116, 118.Similarly, the report generated from the in situ biological images at142 may be refined and finalized by a practitioner at 150, and stored ina database at 152. The patient health data report generated at 144 maybe stored in a patient health data database, at 154.

In some examples, labeling the images at 116 and/or 118 may includeperforming a labeling feedback loop, indicated at 117. Said labelingfeedback loop may be performed by the processing unit and/or by themachine learning algorithm, and may involve performing manual and/orsemi-supervised labeling on images that have already been labeled.Performing the labeling feedback loop at 117 may improve trainingefficiency of the machine learning algorithm, and/or enable training ofthe machine learning algorithm using big data.

With reference to FIG. 12, further details of preparing data fortraining machine learning algorithm 11 at 104 are described. Thedatabase where the reports about the in situ biological images arestored at 152 (FIG. 11) may be filtered and extracted at 156. Forexample, the database may be filtered for all relevant procedures, basedon relevant features, for training machine learning algorithm 11. Forexample, the database may be filtered based on demographic information,weight, body mass index, habits such as alcohol consumption, smoking,and drug use, diet (e.g., read meat, fruit, and vegetable consumption),medications (e.g., nonsteroidal anti-inflammatory drugs, aspirin use,steroids, immunosuppressants, serotonin reuptake inhibitors,chemotherapy, etc.), familial medical history (e.g., history of coloncancer, acid reflux, lynch syndrome, heart disease, adenomatouspolyposis, breast cancer genes), medical history (e.g., depression,heartburn, dyslipidemia), laboratory abnormalities (e.g., hemoglobin,BUN, LDL, troponin, MCV, differentiation of white blood cells), surgicalhistory (e.g., cholecystectomy, polypectomy, colectomy, Nissanfundoplication, bariatric surgery), time of procedure, time since lastprocedure, bowel preparation, blood work results (e.g., red blood cellshape/count/distribution may be informative with respect to bleedingand/or cancer risk), and/or data from an endoscopy or other procedure(e.g., diagnosis code, phrasing, polyp description, time of day, etc.).Raw text may be extracted from a report generated by such filtering(step 156), at 158, and images may be extracted from the report at 160.Such text data extraction at 158 and image extraction at 160 arecomponents of the data extraction step 112 of FIG. 9, and thusextracting text data at 158 and extracting images at 160 involveextracting text data and images from medical reports acquired at 110(FIG. 9) and/or at 130 (FIG. 10).

Data may be aggregated from multiple sources and/or times, at 162, suchas from the patient health data database generated at 154 (FIG. 11). Forexample, data may be aggregated at 162 that includes both real-time dataand historical or previously collected data pertaining to the patient.Such aggregated data may be combined with the raw text extracted at 158,and then parsed and cleaned up, at 164. A master database may be createdand loaded at 166, containing text data. As used herein “text data” mayinclude text, numerical data, dates, times, etc. In parallel, the imagesextracted at 160 may be converted to different color spaces at 168, insome examples. For example, color images may be converted to greyscale,HSV, and etc. Finally, each in situ biological image may be indexed andassigned a sequential image index at 170. Said sequential image indextypically will be based chronologically, though other indexes may beused in other examples. Image data from the indexing at 170 may belabeled at 116, 118 (FIGS. 9 and 11).

FIG. 13 provides some details regarding data de-identification at 114(FIG. 9). The master database of text data created at 166 (FIG. 12) maybe subjected to de-identification before being used to train machinelearning algorithm 11. In some examples, additional patient health datamay be computed at 169. For example, the patient's age may be computedrather than relying on an entered age for the patient in case mistakesare present. Methods 100 also may include feature elimination, at 171,in which unnecessary features or information are removed from the masterdatabase, according to the particular application of machine learningalgorithm 11. In some examples, the feature elimination at 171 isperformed manually, such that a practitioner or operator of the machinelearning algorithm selects which features to remove from the masterdatabase at 171. Additionally or alternatively, input such as featuresensitivity analysis may be received from training the machine learningalgorithm at 172 to inform which features should be eliminated at 171.In other words, feature elimination at 171 may be an automatedelimination of features determined to be unnecessary by machine learningalgorithm 11. The feature elimination at 171 may enable machine learningalgorithm 11 to require fewer computing resources and/or increaseprocessing speed. Additionally or alternatively, identification ofsalient parameters, features, and/or thresholds that are more importantto decision-making in making medical condition state determinations(e.g., the feature elimination at 171) can improve processing speedand/or reduce network latency of machine learning algorithm 11. A UUIDmay be assigned to each individual or patient represented in thedataset, at 174, and thus a de-identified master database of data (e.g.,text data) is completed at 176.

Methods of preparing and aggregating data at 104 and de-identifying dataat 114 shown in FIGS. 12-13 may be automated, such that they areautomatically performed by processing unit 12 and/or machine learningalgorithm 11, via software programmed therein. For example, processingunit 12 and/or machine learning algorithm 11 may be programmed toautomatically extract historically collected procedural images and textfields from PDF files (e.g., at 112, 158, 160 in FIG. 12), link thisdata with other historical patient health records (e.g., step 162 inFIG. 12), such as from the associated pathology reports, aggregate allthe data into a single database (e.g., at 166 in FIG. 12) such as via amedical record identifier, de-identify personally identifiable datafeatures (e.g., at 174 in FIG. 13), generate a complete de-identifieddatabase and identified database with linked image artifacts (e.g., at176 in FIG. 13), keeping the de-identified database and linked artifactsphysically separated to enable protection of patient privacy, and thenuse the de-identified data to train presently disclosed machine learningalgorithms 11. Deidentifying the data at 114 may enable HIPAA data to betransformed into a protected format that can be accessed by individualsto support for supervised and semi-supervised labeling, while protectingpatient confidentiality.

With reference to FIG. 14, the image data that was indexed at 170 (FIG.12) and/or labeled at 116, 118 (FIGS. 9 and 11) is input to machinelearning algorithm 11, at 178, along with the text data from thede-identified database (at 176 in FIG. 13), at 180. Thus, methods 106 oftraining machine learning algorithm 11 use both image data andcorresponding text data to train machine learning algorithm 11 to makemedical condition state determinations, using the text-based databaseand the at least one labeled biological image. Thus, training machinelearning algorithms 11 at 106 ultimately includes acquiring data from atleast one medical procedure (e.g., performing the procedure at 124 inFIG. 10), including acquiring at least one in situ biological image ofan area of a patient's body (at 128 in FIG. 10) and acquiring one ormore biological specimens from the area (at 126 in FIG. 10). Theseimages and text data are input into machine learning algorithm 11 at178, 180, respectively, after the intervening steps of acquiring thedata (at 102 in FIGS. 9-10), preparing the data (at 104 in FIGS. 9 and11-13) and aggregating the data (at 162 in FIG. 12). Training machinelearning algorithm 11 at 106 thus also includes labeling the at leastone in situ biological image (at 138, 140 in FIG. 10, thereby creatingat least one labeled biological image that indicates respective medicalcondition states shown in each respective biological image. Generally, aplurality of in situ biological images will have been labeled, such thatmachine learning algorithm 11 receives a plurality of labeled imageswhen the image data is input at 178. Training machine learning algorithm11 at 106 also includes acquiring patient health data pertaining to thepatient from a plurality of data sources (at 130 in FIG. 10),aggregating the patient health data acquired from the plurality of datasources into a text-based (or other format) database (at 162 in FIG.12), and de-identifying the patient data in the text-based database (at114 in FIGS. 9 and 13).

Methods of training the machine learning algorithm at 106 also mayinclude selecting relevant features from the text-based database ofpatient health information, at 182. For example, in some examples, onlya subset of the categories of patient health data in the master databasewill be relevant or needed for a particular type of medical conditionstate determination. This selection at 182 may be performed by themachine learning algorithm (e.g., as a result of training and therebylearning which features may not be necessary), by the processing unit,and/or manually by a practitioner or other user responsible forpreparing and training the machine learning algorithm. Once the relevantfeatures are selected at 182, unnecessary features may be eliminated at184, such as by removing the unneeded features from the database.Information about the relevant features that were selected at 182 alsomay be fed back into the system during later de-identification in someexamples, as indicated at 172 in FIG. 13. Training the machine learningalgorithm at 106 also may include testing, training, and/or validatingthe algorithm at 186, and finally, deploying the machine learningalgorithm at 108. Training and testing the machine learning algorithm at186 is generally limited to offline learning, though in some examplesonline learning may be performed by updating the machine learningalgorithm's neural network weights, such that the machine learningalgorithm continues to learn as it receives additional image input andpatient health data during training. In some examples, training andtesting the machine learning algorithm at 186 includes splitting,separating, or partitioning, a dataset into three or more subsets to beused in different phases of training the machine learning algorithm. Forexample, a given dataset may be separated into a training dataset, atesting dataset, and a validation dataset. In this example, training andtesting the machine learning algorithm at 186 may include training themachine learning algorithm using the training dataset, testing themachine learning algorithm, using the testing dataset, and validatingthe machine learning algorithm using the validation dataset. In thismanner, different data within a given dataset may used in differentphases of training and testing the machine learning algorithm at 186. Ina specific example, a majority of a dataset (e.g., greater than 50%,greater than 60%, greater than 70%, and/or greater than 80% of the datain a dataset) may be used as the training dataset, while a smallerproportion of the dataset may be reserved for the testing dataset andfor the validation dataset.

FIG. 15 illustrates other methods 106 of training machine learningalgorithm 11 to make medical condition state determinations, accordingto the present disclosure. Examples of training the machine learningalgorithm 106 shown in FIG. 15 are not meant to be exclusive, and mayoverlap with and/or be combined with other methods of training machinelearning algorithm 106, described herein. An image input (e.g., imageinput 14) may be received by at least one processing unit (e.g.,processing unit 12) at 200, with the image input being one or moreimages from an imaging device (e.g., imaging device 16). The at leastone processing unit includes the machine learning algorithm stored inone or more memories, though in other examples, the processing unitreceiving the image input may be separate from the machine learningalgorithm. Receiving the image input at 200 may include receivinglabeling information for the image input, such as manual-derived orsemi-supervised-derived labeling information (e.g., from 116, 118 inFIGS. 9 and 11).

Methods of training the machine learning algorithm at 106 shown in FIG.15 also include receiving patient health data (e.g., patient health data18) as input at 202, with the receiving patient health data at 202 alsobeing performed by the at least one processing unit. In some examples,receiving the patient health data at 202 includes collecting and/orretrieving the patient health data, and delivering the patient healthdata to the at least one processing unit. For example, a practitionermay collect and enter patient health data at 202 from a chart or medicalrecords, from asking the patient for information, and then entering orinputting the patient health data into the processing unit. In someexamples, collecting and/or retrieving the patient health data may beperformed in real-time. For example, patient health data may be obtainedand entered during a colonoscopy procedure, such as by asking a patientquestions during the procedure. Additionally or alternatively,collecting and/or retrieving patient health data may be performed beforethe procedure (e.g., before the colonoscopy), and therefore beforereceiving image input for analysis by the machine learning algorithm.For example, patient health data collected and stored in medical recordsgenerally will have been collected prior to the medical procedure, andmay be accessed before or during the procedure and input to the machinelearning algorithm. In various examples, patient health data receivedand/or collected at 202 may include survey question answers, staticdata, electronic health records, electronic medical records, demographicinformation, medications, drug use, smoking history, computed riskpredictors, blood work, prior procedural results, and/or risk factors.As specific examples, patient health data collected and/or received at202 may include body mass index (BMI), physical activity, cigarettesmoking history, alcohol usage, family history, inflammatory boweldisease, current hormone therapy (e.g., postmenopausal hormone therapy),former hormone therapy (e.g., postmenopausal hormone therapy), aspirinusage, nonsteroidal anti-inflammatory drugs (NSAIDs) usage, consumptionof processed and/or red meat, fruit and vegetable consumption levels,demographic information, medications, drug usage, diet type and quality,dietary fat intake, weight, height, age, race, presence of otherillnesses, biological markers, INR/PTT/PT/platelets/bleed time, previousendoscopy results, previous CT scan results, previous angiograminformation, previous MRI results, and/or previous sonography data.

In some examples, receiving the patient health data at 202 may includede-identifying the patient health data (e.g., at 114 from FIGS. 9 and13). Said data de-identification generally will be performed during thetraining the machine learning algorithm at 106, and are optional duringthe deploying the machine learning algorithm at 108. In other words,patient health data input into the machine learning algorithm may bede-identified if the data is being used to train the machine learningalgorithm, but patient-identifying information may be retained inassociation with the patient health data when the machine learningalgorithm is deployed to make a medical condition state determinationfor a particular patient. There may be some examples where it is desiredto train the machine learning algorithm using data that has not beende-identified, or where it is desired to de-identify the data duringdeployment of the machine learning algorithm as well.

Training the machine learning algorithm 106 also may include encodingthe patient health data at 204, thereby converting the patient healthdata to encoded patient health data (e.g., encoded patient health data20). The encoding the patient health data and converting the patienthealth data at 204 is performed by the at least one processing unit,according to instructions stored on the processing unit. The encodedpatient health data is then embedded into at least one image of theimage input, at 206. Thus, disclosed systems are configured such thatthe machine learning algorithm is informed and trained using bothtraining images (e.g., image input) and patient health training datathat is encoded such that it may be added to, appended to, overlaid on,and/or embedded in the image training data.

In some examples, embedding the encoded patient health data at 206includes embedding the encoded patient health data within a consistentregion of the at least one image, such as described in connection withFIGS. 7-8. The embedding the encoded patient health data is performed bythe at least one processing unit, wherein the machine learning algorithmis configured to make a medical condition state determination based onthe image input and the encoded patient health data. To do so, the imageinput and the embedded patient health data is input to (and received by)the machine learning algorithm for analysis and processing at 208. Insome examples, the encoded patient health data is input into a fullyconnected network portion of the machine learning algorithm. The machinelearning algorithm thus may be trained at 106 using this encoded andembedded patient health data, along with the image input.

Training and preparing the machine learning algorithm at 106 in FIG. 15also may be understood from the perspective of programming theprocessing unit to perform the steps described above. For example,receiving the image input at 200 may include programming at least oneprocessing unit (e.g., processing unit 12) to receive an image input(e.g., image input 14), with the image input being one or more imagesfrom an imaging device (e.g., imaging device 16), and with the machinelearning algorithm being stored within a memory of the processing unit.In other examples, the machine learning algorithm may be accessed by theprocessing unit, rather than stored therein. Similarly, receiving thepatient health data at 202 may include programming the processing unitto receive patient health data as input, and encoding the patient healthdata at 204 may include programming the at least one processing unit toencode the patient health data and thereby convert the patient healthdata to encoded patient health data. Embedding the encoded patienthealth data at 206 may include programming the processing unit to embedthe encoded patient health data into at least one image of the imageinput. In this manner, the processing unit may be programmed such thatit is configured to make a medical condition state determination, viathe machine learning algorithm, based on the image input and the encodedpatient health data.

Once trained, the machine learning algorithm may be deployed at 108according to the same methods, where image input is received at 200,patient health data is received at 202, the patient health data isencoded at 204 and embedded at 206, and then input to the machinelearning algorithm for analysis and processing at 208, to thus make amedical condition state determination using the image input and patienthealth data. Thus, FIG. 15 may represent methods of training the machinelearning algorithm at 106 and methods of deploying the machine learningalgorithm at 108. In the deploying at 108, receiving the image input at200 may include accessing an image input by the machine learningalgorithm, again with the image input being one or more images from animaging device used to perform a medical imaging procedure on a patient.Similarly, receiving the patient health data at 202 may includeaccessing and/or retrieving patient health data with the machinelearning algorithm. Accessing and/or retrieving the patient health dataat 202 may be performed in real-time, and/or may include accessing orretrieving patient health data that was collected or provided before theaccessing the image input at 200 (e.g., before the medical procedure).Analyzing and processing the image input and encoded patient health dataat 208 may include causing the machine learning algorithm to analyze theimage input and the patient health data together to make the medicalcondition state determination. In such methods, the machine learningalgorithm may be configured to encode the patient health data to convertthe patient health data to encoded patient health data at 204, and/orsuch encoding at 204 may be performed by the processing unit. Similarly,the embedding the encoded patient health data at 206 may be performed bythe machine learning algorithm and/or by the processing unit. Themachine learning algorithm makes a medical condition state determinationat 218, based on its analyzing and processing of the image input andencoded patient health data at 208. The analysis results (e.g., themedical condition state determination) produced by the machine learningalgorithm may be accessed at 220, such as via the output image displayedat 220, which serves as a visual representation of the encoded patienthealth data viewable on the analysis results, all together as a singlesource (e.g., visual output augmented with encoded patient health dataand the medical condition state determination in the output image).While prior art machine learning models were not able to providevisualization of data from multiple sources to a physician or otherpractitioner, currently disclosed machine learning algorithms 11 areable to provide this functionality.

In training the machine learning algorithm at 106 and/or deploying themachine learning algorithm at 108, the encoded patient health data maybe embedded into the at least one image of the image input at 206 at orbefore a time that the machine learning algorithm analyzes and processesthe image input at 208, such that the machine learning algorithmanalyzes the image input together with the encoded patient health dataembedded in the at least one image of the image input. In other words,the machine learning algorithm may be trained to analyze the patienthealth data integrally with the image input being analyzed. In variousexamples of training at 106 and deploying at 108, encoding the patienthealth data at 204 and/or encoding dynamic state information at 212(discussed below) may include performing a one-hot encoding conversionand/or performing data dictionary encoding. Additionally oralternatively, encoding the patient health data at 204 may includeconverting the patient health data to a plurality of collections ofcoded image pixels (e.g., collection of coded image pixels 34) that areadded to, appended to, overlaid on, and/or embedded in the at least oneimage of the image input.

In some examples, embedding the patient health data at 206 may includeadding, appending, and/or overlaying the encoded patient health data tothe image input as a vector and/or a data tensor (though as used herein,a “vector” is a subset of, or type of, tensor (also known as a “datatensor”), wherein the adding, appending, and/or overlaying generallybeing performed by the at least one processing unit. Additionally oralternatively, the encoded patient health data may be added, appending,and/or overlaid onto a data tensor. Encoding the patient health data at204 may include selectively representing the patient health data and/ordynamic state information as a tensor and/or as a plurality ofcollections of coded image pixels, in various examples of presentlydisclosed systems.

Some examples of methods of training the machine learning algorithm at106 and/or deploying the machine learning algorithm at 108 includeperforming similar steps with dynamic state information (e.g., dynamicstate information 48), in addition to the patient health data. Forexample, dynamic state information may be received by the processingunit at 210 (e.g., from one or more dynamic state apparatus 46configured to deliver said dynamic state information as additionalinput), encoded at 212 to form encoded dynamic state information (e.g.,encoded dynamic state information 49), and embedded into the at leastone image of the image input at 214, at or before a time that themachine learning algorithm analyzes the image input. In this manner, theencoded dynamic state information also may be input to the machinelearning algorithm, such that the machine learning algorithm may analyzethe image input together with the encoded dynamic state information (andthe encoded patient health data) embedded in the at least one image ofthe image input, at 208. In some examples, the encoded dynamic stateinformation is input into a fully connected network portion of themachine learning algorithm during the analyzing at 208. In someexamples, embedding the encoded dynamic state information at 214includes adding, appending, and/or overlaying encoded dynamic stateinformation to the image input as a vector and/or a data tensor, withthe appending generally being performed by the at least one processingunit.

In some examples, the encoded patient health data may be embedded intothe image input at 206 before the image input is received by theprocessing unit at 200. In other words, a processing unit other than theprocessing unit that includes the machine learning algorithm may performthe encoding at 204 and/or the embedding at 206, in some examples.Similarly, and additionally or alternatively, the encoded dynamic stateinformation may be embedded into the image input at 214 before the imageinput is received by the processing unit at 200, such as in exampleswhere a processing unit other than the processing unit that includes themachine learning algorithm performs the encoding at 212 and/or theembedding at 214.

Some examples of methods of training the machine learning algorithm at106 and/or deploying the machine learning algorithm at 108 include imageencoding the encoded patient health data and/or image encoding theencoded dynamic state information at 216, which may be performed by theprocessing unit after encoding the patient health data at 204.Additionally or alternatively, methods 106, 108 may include imageencoding the dynamic state information at 216, which may be performed bythe processing unit after encoding the dynamic state information at 212.Image encoding the encoded patient health data and/or the encodeddynamic state information at 216 may include adding, overlaying, and/orappending image pixels to the image input. Said image encoding at 216,if performed, is performed prior to inputting the image input to themachine learning algorithm for analysis and processing at 208. In someexamples, the image encoding at 216 may be performed prior to thereceiving the image input at 200, such as in examples where a differentprocessing unit performs the image encoding than the processing unitthat includes the machine learning algorithm.

Methods of training the machine learning algorithm at 106 and/ordeploying the machine learning algorithm at 108 include making a medicalcondition state determination at 218, such as by detecting, classifying,and/or localizing a feature of interest in one or more image inputsbased on the image input and the encoded patient health data. Making themedical condition state determination generally includes processing andinterpreting the encoded patient health data along with the image inputitself. Again, the encoded patient health data is embedded in at leastone image of the image input, such as in the form of a plurality ofcollections of coded image pixels added to the image input. Making themedical condition state determination at 218 may be performed in realtime. In other words, disclosed machine learning algorithms may be usedto make medical condition state determinations while the medicalprocedure is being performed to produce the image input, with saidmedical condition state determination being based on the image input andthe encoded patient health data. For example, images from a colonoscopyprocedure may be sent to the machine learning algorithm during thecolonoscopy, and the machine learning algorithm may be configured todetect, classify, and/or localize polyps and/or other medical conditionstates or features in real-time during the colonoscopy procedure orother medical procedure. Making a medical condition state determinationat 218 generally will be an automated determination, or at least asemi-automated determination, by the machine learning algorithm.Furthermore, making the medical condition state determination at 218 mayinclude determining a probabilistic diagnosis (e.g., a confidence level,which may be expressed in the form of a percentage) of the medicalcondition state of the image input, via the machine learning algorithm,and/or any additional relevant information, such as the informationdiscussed above in connection with FIGS. 3-6.

Methods of training the machine learning algorithm at 106 and/ordeploying the machine learning algorithm at 108 may include accessingthe results and/or displaying information (e.g., the medical conditionstate determination) at 220 after the machine learning algorithm hasanalyzed the image input, the encoded patient health information, and/orthe dynamic state information to make the medical condition statedetermination. For example, displaying information at 220 may includeproducing and displaying an output image (e.g., output image 40) on adisplay device (e.g., display device 42), which generally will show theimage input that includes the medical condition that was detected by themachine learning algorithm (e.g., visual output for the medicalcondition state determination), information about the medical conditionstate determination (e.g., diagnosis and confidence in thedetermination), along with at least some relevant patient health datapertaining to the medical condition state determination (e.g., encodedpatient health data 20). In this manner, disclosed systems produce anddisplay output images at 220 that show visual output that is augmentedwith encoded patient health data. Producing the output image at 220 maybe performed by the machine learning algorithm and/or by the processingunit (e.g., an encoding algorithm of processing unit 12) described inconnection with disclosed systems 10. In some examples, displayinginformation at 220 includes displaying the encoded patient health datawithin a consistent region of an output image (e.g., within a givenregion 32), wherein the displaying is performed by, or instructed by,the at least one processing unit. In some examples, displayinginformation at 220 includes displaying real-time patient health historydata. In some examples, displaying information at 220 includesdisplaying the encoded patient health data via labels and/or icons(e.g., icons 44).

Turning now to FIGS. 16-17, the architecture and operation of examplesof machine learning algorithms 11 according to the present disclosureare described. As shown, in one example of machine learning algorithm11, image inputs 14 are input into machine learning algorithm 11, suchas to a standard convolutional neural network 60 of machine learningalgorithm 11. Standard convolutional neural network 60 may be fullytrained with randomly initialized weights, or may be a transfer learningneural network with pre-trained weights. In some examples, transferlearning can enable machine learning algorithm 11 to be trained usingsmaller images and/or less data, though fully trained models on largerdatasets with medically relevant images may be more accurate forreal-time medical condition state determinations according to thepresent disclosure. Machine learning algorithm 11 may include areshaping layer 62 and one or more fully connected (dense) layers 64. Invarious examples of systems 10, patient health data 18 may input intomachine learning algorithm 11 at one or more different points. Forexample, processing unit 12 may perform an encoding operation on patienthealth data 18 (which may be a one-hot encoding conversion, and/or adifferent type of encoding) to produce encoded patient health data 20.In some examples, encoded patient health data 20 may be input intomachine learning algorithm 11 by inputting encoded patient health data20 directly into neural network 60, as indicated by arrow 66.Additionally or alternatively, encoded patient health data 20 may beinput into machine learning algorithm 11 along with image input 14, asindicated by arrow 68. Similarly, encoded dynamic state information 49may be input into machine learning algorithm 11 by inputting encodeddynamic state information 49 directly into neural network 60, also asindicated by arrow 66, and/or encoded dynamic state information 49 maybe input into machine learning algorithm 11 along with image input 14,as also indicated by arrow 68.

In some examples, this process involves adding (e.g., appending,overlaying, and/or embedding) encoded patient health data 20 and/orencoded dynamic state information 49 to image input 14 as a vector ordata tensor (arrow 66). For example, processing unit 12 may beprogrammed to add encoded patient health data 20 to image input 14 as avector after image input 14 is reshaped (e.g., flattened) and/orconcatenated, as represented by reshaping layer 62. For example, thereshaping operation may be configured to convert tensor data into vectordata and then added (e.g., appended and/or concatenated) to image input14 within the layers of machine learning algorithm 11 and/or added toimage input 14 before being input into machine learning algorithm 11.Similarly, encoded dynamic state information 49 also may be appended toimage input 14 as a vector or data tensor, such that processing unit 12may be configured to add encoded dynamic state information to the imageinput as the vector or the data tensor after the image input is reshaped(e.g., flattened) and/or concatenated. In other words, processing unit12 may be configured to embed encoded patient health data 20 and/orencoded dynamic state information 49 into a tensor of machine learningalgorithm 11. In some examples, vector data representing encoded patienthealth data 20 and/or encoded dynamic state information 49 may be addedto image input within a fully connected network portion of machinelearning algorithm 11 (e.g., adding the vector to fully connected layers64).

In examples that include inputting encoded patient health data 20 and/orencoded dynamic state information 49 with image input 14 (arrow 68),collections of coded images pixels (e.g., collection of coded imagepixels 34) may be appended to image input 14 before image input 14 isinput to machine learning model 11, as indicated by arrow 68. In someexamples, image encoding is performed (indicated at 70) before the imagepixels are appended to image input 14. Image encoding at 70 may includeoverwriting pixels on top of the original image input and/or expandingthe image size or resolution, such as by adding pixels to the border.

FIG. 17 illustrates similar examples of system 10 with small variations.In some examples represented in FIG. 17, patient health data 18 (whichmay be static data and/or meta-data) is encoded at 204, thereby beingconverted to encoded patient health data 20, and dynamic stateinformation 48 is encoded at 212, thereby being converted to encodeddynamic state information 49, which may be real-time sensor-derived datainput. In some examples, encoded patient health data 20 and/or encodeddynamic state information 49 is input into machine learning algorithm 11by appending vector data to the image input (which may have beenreshaped into a tensor) within the neural network (as indicated byarrows 66 and 72). In these examples, machine learning algorithm 11 maybe a custom, or atypical, convolutional neural network architecture.Additionally or alternatively, encoded patient health data 20 and/orencoded dynamic state information 49 is optionally subjected to imageencoding at 70 and embedded into image input 14 prior to the image inputbeing input to the neural network, such as by appending image pixels toimage input 14 (indicated by arrow 68). In these examples, machinelearning algorithm 11 may be a generic, or standard, neural network.

FIG. 18 shows another view of examples of machine learning algorithm 11of systems 10. As shown in FIG. 18, machine learning algorithm mayinclude one or more convolutional layers 74 and one or more fullyconnected layers 64. As with other examples of system 10, one or moreimage inputs 14 are input into machine learning algorithm 11. Eachconvolutional layer 74 may transform, or reshape, dimensions of imageinput 14, until the image is reshaped (e.g., flattened) into a tensor 76or a vector 76 to be operated on in subsequent fully connected layers64. Tensor 76 may be transformed into one or more other tensors 78 fromwithin the fully connected layer(s) 64 of the neural network beforeoutput image 40 is produced with the medical condition state determinedby machine learning algorithm 11. As shown in other examples, patienthealth data 18 and/or dynamic state information 48 may be reshapedand/or encoded at 204 and/or 212 to produce vector data 80, whichrepresents an example of encoded patient health data 20 and/or encodeddynamic state information 49. Said vector data 80 may be added to tensor76 or tensor 78 (which are tensor representations of image input 14) bya concatenation operation indicated at 82. In some examples of system10, concatenation 82 occurs before passing data from convolutionallayers 74 into fully connected layers 64 and after flattening or areshape operation such that the tensor dimensionality matches to permita concatenation operation.

In some examples of system 10, concatenation 82 occurs within fullyconnected layers 64, as represented by vector data 80 being shownconcatenated on tensor 76 and tensor 78. This concatenation 82 occursbefore machine learning algorithm 11 makes a medical condition statedetermination and before output image 40 is produced, such that machinelearning algorithm 11 is informed by both image input 14 (which has beenconverted to the form of tensors 76, 78) and encoded patient health data20, which is encoded as vector data 80 concatenated onto tensors 76,78within fully connected layers 64. The machine learning algorithm 11makes a medical state determination and is then passed to an outputimage rendering function which facilitates creation of a composite image(e.g., output image 40) which will visualize the medical statedetermination state information in the desired configuration and whichmay be rendered on a display device (e.g., display device 42), asdescribed in detail herein. Output image 40 may include informationabout the classification, localization, and/or confidence level of themedical condition state determination made by machine learning algorithm11. For example, output image 40 may display a bounding box (e.g.,bounding box 36) with a box width, box height, and/or a classificationand associated confidence level. Saud output image 40 may be rendered(e.g., by image display rendering function 41), displayed (e.g., ondisplay device 42), printed, and/or at least partially communicated viaauditory signals in various examples according to the presentdisclosure.

Illustrative, non-exclusive examples of inventive subject matteraccording to the present disclosure are described in the followingenumerated paragraphs:

A1. A system for preparing, training, and deploying a machine learningalgorithm for making a medical condition state determination, the systemcomprising:

at least one processing unit comprising the machine learning algorithm,wherein the machine learning algorithm is stored in one or more memoriesof the at least one processing unit, wherein the at least one processingunit is programmed to:

-   -   receive an image input from an imaging device, wherein the image        input comprises one or more images obtained and/or produced by        the imaging device;    -   receive patient health data as input;    -   encode the patient health data to convert the patient health        data to encoded patient health data; and    -   transmit the encoded patient health data into the machine        learning algorithm,

wherein the system is configured to make the medical condition statedetermination based on the image input and the encoded patient healthdata, via the machine learning algorithm.

A1.1 The system of paragraph A1, wherein the system is configured suchthat the encoded patient health data is embedded into at least one imageof the image input at or before a time that the machine learningalgorithm analyzes the image input, such that the machine learningalgorithm analyzes the image input together with the encoded patienthealth data embedded in the at least one image of the image input.

A1.2. The system of paragraph A1 or A1.1, further comprising the imagingdevice, wherein the imaging device is configured to produce the one ormore images.

A1.3. The system of any of paragraphs A1-A1.2, wherein the machinelearning algorithm comprises a convolutional neural network.

A1.4. The system of any of paragraphs A1-A1.3, wherein the machinelearning algorithm comprises a transformer, an LSTM, an RNN, an SVM, adense neural network, an auto-encoder, and/or a YOLO.

A1.5. The system of any of paragraphs A1-A1.4, wherein the at least oneprocessing unit is configured to receive the patient health data as aninput tensor.

A1.6. The system of any of paragraphs A1-A1.5, wherein the at least oneprocessing unit is further configured to receive a sound input, an NIRinput, sensor measurement data, 2D vector data, 3D vector data, and/or4D vector data.

A1.7. The system of any of paragraphs A1-A1.6, wherein the at least oneprocessing unit is configured to embed the encoded patient health datainto at least one image of the image input.

A1.8. The system of any of paragraphs A1-A1.7, wherein the at least oneprocessing unit is configured to embed the encoded patient health datainto a tensor of the machine learning algorithm.

A1.9. The system of any of paragraphs A1-A1.8, wherein the machinelearning algorithm is trained using one or more selected from the groupconsisting of unsupervised learning, semi-supervised learning, andsupervised learning.

A2. The system of any of paragraphs A1-A1.9, wherein the machinelearning algorithm comprises a fully trained convolutional neuralnetwork.

A3. The system of any of paragraphs A1-A2, wherein the machine learningalgorithm comprises a transfer learning convolutional neural network.

A4. The system of any of paragraphs A1-A3, wherein the at least oneprocessing unit is further programmed to perform one-hot encodingconversion to encode the patient health data and thereby convert thepatient health data to the encoded patient health data.

A5. The system of any of paragraphs A1-A4, wherein the system furthercomprises a one-hot encoding system configured to convert dataconcerning patient risk factors into a numerical representation based ona predefined data schema, thereby converting the patient health data tothe encoded patient health data.

A6. The system of paragraph A4 or A5, wherein the at least oneprocessing unit is further programmed to image encode the patient healthdata after one-hot encoding conversion of the patient health data.

A7. The system of any of paragraphs A1-A6, wherein the at least oneprocessing unit is further programmed to perform a reshaping and/orconcatenating operation to reshape, flatten, and/or concatenate theencoded patient health data and/or at least one image of the imageinput.

A8. The system of any of paragraphs A1-A7, wherein the system isconfigured to display the encoded patient health data within a region ofat least one image of the image input.

A8.1. The system of any of paragraphs A1-A8, wherein the system isconfigured to embed the encoded patient health data within a region ofat least one image of the image input.

A9. The system of any of paragraphs A1-A8.1, wherein the encoded patienthealth data comprises a plurality of collections of coded image pixelsthat are added to the image input.

A9.1. The system of paragraph A9, wherein the plurality of collectionsof coded image pixels are appended to the image input.

A9.2. The system of paragraph A9, wherein the plurality of collectionsof coded image pixels are overlaid onto to the image input.

A10. The system of any of paragraphs A1-A9.2, wherein the at least oneprocessing unit is further programmed to add the encoded patient healthdata to the image input as a vector and/or a data tensor.

A10.1. The system of any of paragraphs A1-A10, wherein the at least oneprocessing unit is further programmed to add dynamic state informationto the image input as a/the vector and/or a/the data tensor.

A10.2. The system of any of paragraphs A1-A10.1, wherein the at leastone processing unit is configured to append the encoded patient healthdata to a/the data tensor.

A11. The system of paragraph A10 or A10.1, wherein the at least oneprocessing unit is further programmed to add the encoded patient healthdata to the image input as the vector or the data tensor after the imageinput is reshaped and/or concatenated.

A11.1. The system of any of paragraph A10-A11, wherein the at least oneprocessing unit is further programmed to add encoded dynamic stateinformation to the image input as the vector and/or the data tensorafter the image input is reshaped and/or concatenated.

A12. The system of any of paragraphs A1-A11.1, wherein the system isconfigured to selectively represent the patient health data as a/thetensor and/or a/the plurality of collections of coded image pixels.

A13. The system of any of paragraphs A1-A12, wherein the machinelearning algorithm is configured to receive the encoded patient healthdata into a fully connected network portion of the machine learningalgorithm.

A14. The system of any of paragraphs A1-A13, wherein the system isconfigured to perform real-time medical condition state determination.

A14.1. The system of any of paragraphs A1-A14, wherein the system isconfigured to perform automated medical condition state determination.

A14.2. The system of any of paragraphs A1-A14.1, wherein the system isconfigured to automatically generate a report for a/the patient thatincludes a summary of the medical condition state determination, alongwith billing information for the procedure.

A15. The system of any of paragraphs A1-A14.2, further comprising acomputing device configured for collecting and/or retrieving the patienthealth data, wherein the computing device is further configured todeliver the patient health data to the at least one processing unit.

A16. The system of paragraph A15, wherein the computing device isconfigured to collect and/or retrieve the patient health data inreal-time from a database.

A16.1. The system of paragraph A16, wherein the database compriseselectronic health records and/or electronic medical records.

A16.2. The system of any of paragraphs A1-A16.1, wherein the patienthealth data is collected or provided before the image input is providedto the at least one processing unit.

A17. The system of any of paragraphs A1-A16.2, wherein the patienthealth data comprises survey question answers, static data, active data,the electronic health records, the electronic medical records, and/orrisk factors.

A17.1. The system of any of paragraphs A1-A17, wherein the patienthealth data comprises body mass index (BMI), physical activity,cigarette smoking history, alcohol usage, family history, presence ofinflammatory bowel disease, current hormone therapy (e.g.,postmenopausal hormone therapy), former hormone therapy (e.g.,postmenopausal hormone therapy), aspirin usage, nonsteroidalanti-inflammatory drugs (NSAIDs) usage, consumption of processed and/orred meat, fruit and vegetable consumption levels, demographicinformation, medications, drug usage, diet type and quality, dietary fatintake, weight, height, age, race, presence of other illnesses,biological markers, INR/PTT/PT/platelets/bleed time, previous endoscopyresults, previous CT scan results, previous angiogram information,previous MRI results, computed risk predictors, blood work, priorprocedural results, and/or previous sonography data.

A18. The system of any of paragraphs A1-A17.1, wherein the machinelearning algorithm is configured to detect one or more medical conditionstates based on the one or more images and the patient health data.

A18.1. The system of any of paragraphs A1-A18, wherein the system isconfigured to display the one or more medical condition states and/orthe patient health data.

A18.2. The system of paragraph A18.1, wherein the system is configuredto display the one or more medical condition states and/or the patienthealth data using a bounding box, text, a shaped outline, a visualindication on a screen or monitor, and/or an auditory signal.

A19. The system of any of paragraphs A1-A18.2, wherein the machinelearning algorithm is configured to classify a/the one or more medicalcondition states based on the one or more images and the patient healthdata.

A20. The system of any of paragraphs A1-A19, wherein the machinelearning algorithm is configured to localize a/the one or more medicalcondition states based on the one or more images and the patient healthdata.

A21. The system of any of paragraphs A1-A20, wherein the at least oneprocessing unit is configured to perform de-identification of thepatient health data.

A22. The system of any of paragraphs A1-A21, wherein the at least oneprocessing unit is configured to receive manual labeling information forthe image input.

A23. The system of any of paragraphs A1-A22, wherein the at least oneprocessing unit is configured to receive semi-supervised labelinginformation for the image input.

A23.1. The system of any of paragraphs A1-A23, wherein the at least oneprocessing unit is configured to first receive the supervised labelinginformation, and then to receive the semi-supervised labelinginformation, and then to perform auto-labeling.

A23.2. The system of any of paragraphs A1-A23.1, wherein the at leastone processing unit is configured to first receive the supervisedlabeling information and/or to perform auto-labeling, and then toreceive the semi-supervised labeling information to refine and improvelabels in a dataset.

A24. The system of any of paragraphs A1-A23.2, wherein the system isconfigured to perform a labeling feedback loop comprising thesemi-supervised labeling information.

A24.1. The system of paragraph A24, wherein the labeling feedback loopimproves training efficiency of the machine learning algorithm.

A24.2. The system of paragraph A24 or A24.1, wherein the labelingfeedback loop enables the machine learning algorithm to be trained usinglarge scale data and/or big data.

A25. The system of any of paragraphs A1-A24.2, wherein the system isconfigured to receive large scale data and/or big data for training themachine learning algorithm.

A26. The system of any of paragraphs A1-A25, wherein the machinelearning algorithm comprises a custom convolutional neural network.

A27. The system of any of paragraphs A1-A26, wherein the system isconfigured such that the machine learning algorithm is informed andtrained using both image training data and patient health training datathat is encoded such that it may be appended to, added to, overlaid on,and/or embedded in the image training data.

A28. The system of any of paragraphs A1-A27, wherein the system isconfigured to provide visual output for medical condition statedetermination via a graphical user interface, wherein the visual outputis augmented with the patient health data.

A29. The system of any of paragraphs A1-A28, wherein the patient healthdata comprises information regarding a patient's cardiac cycle and/orbreathing cycle, and wherein the image input is temporally annotatedsuch that each of the one or more images from the imaging device may bematched with a phase of the patient's cardiac cycle and/or breathingcycle.

A30. The system of any of paragraphs A1-A29, wherein the at least oneprocessing unit comprises an encoding algorithm configured to produce anoutput image that comprises at least one image of the image input andthe encoded patient health data.

A30.1. The system of paragraph A30, further comprising a displayconfigured to display the output image.

A30.2. The system of paragraph A30 or A30.1, wherein the output imagecomprises a printed output image and/or a digital output image.

A31. The system of any of paragraphs A30-A30.2, wherein the output imagefurther comprises the medical condition state determination determinedby the machine learning algorithm.

A32. The system of any of paragraphs A1-A31, wherein the system isconfigured for online learning such that the machine learning algorithmis updated using neural network weights, and thereby continues to learnas it receives additional image input and additional patient healthdata.

A33. The system of any of paragraphs A1-A32, wherein the imaging devicecomprises an sonography device, an x-ray device, a computed tomography(CT) scanning device, a magnetic resonance imaging (MRI) device, apositron emission tomography (PET) device, a retinal camera, adermatoscope, a radiograph device, a mammography device, an endoscope, acolonoscopy device, an esophagogastroduodenoscopy device, a bronchoscopydevice, a photoacoustic endoscopy device, an electro-optical sensor, aNBI (Narrow Band Imaging) colonoscopy device, a white light endoscopydevice, a chromoendoscopy device, and/or a capsule endoscopy device.

A34. The system of any of paragraphs A1-A33, further comprising anapparatus for determining dynamic state information of a patient,wherein the apparatus is configured to deliver the dynamic stateinformation to the at least one processing unit as an additional input.

A34.1. The method of paragraph A34, wherein the dynamic stateinformation comprises heart rate, blood pressure, compensated heartrate, anesthetics, telemetry, saline used, oxygen saturation, end tidalcarbon dioxide (capnography), current medications, and/or activity indistal extremities.

A34.2. The method if paragraph A34 or A34.1, wherein the apparatuscomprises one or more motion sensors, one or more accelerometers, and/orone or more video cameras configured for motion detection.

A34.3. The system of any of paragraphs A1-A34.2, wherein the system isconfigured such that dynamic state information is embedded into at leastone image of the image input at or before a time that the machinelearning algorithm analyzes the image input, such that the machinelearning algorithm analyzes the image input together with the dynamicstate information embedded in the at least one image of the image input.

A34.4. The system of any of paragraphs A1-A34.3, wherein the at leastone processing unit is further programmed to perform one-hot encodingconversion to encode the dynamic state information.

A34.5. The system of paragraph A34.4, wherein the at least oneprocessing unit is further programmed to image encode the dynamic stateinformation after one-hot encoding conversion of the dynamic stateinformation.

A34.6. The system of any of paragraphs A1-A34.5, wherein the machinelearning algorithm is configured to receive the encoded dynamic stateinformation into a/the fully connected network portion of the machinelearning algorithm.

A35. The system of any of paragraphs A1-A34.6, wherein the machinelearning algorithm is configured to process and interpret encodedpatient health data that is embedded in at least one image as a/theplurality of collections of coded image pixels.

A35.1. The system of paragraph A35, wherein the plurality of collectionsof coded image pixels comprises a respective collection of coded imagepixels for each respective type or category of encoded patient healthdata.

A35.2. The system of paragraph A35 or A35.1, wherein the plurality ofcollections of coded image pixels are arranged in a row, a column,and/or an array on the at least one image.

A35.3. The system of any of paragraphs A35-A35.2, wherein the pluralityof collections of coded image pixels are positioned together within agiven region of the at least one image.

A35.4. The system of any of paragraphs A35-A35.3, wherein a respectiveshade of each respective collection of coded image pixels represents therelative value of the respective encoded patient health data encoded inthe respective collection of coded image pixels.

A36. The system of any of paragraphs A1-A35.4, wherein the plurality ofcollections of coded image pixels comprises a plurality of collectionsof grayscale-coded image pixels.

A36.1. The system of any of paragraphs A1-A36, wherein the plurality ofcollections of coded image pixels comprises a plurality of collectionsof color-coded image pixels.

A37. The system of any of paragraphs A35-A36.1, wherein the system isconfigured to display a respective icon for each respective collectionof coded image pixels to indicate what the respective collection ofcoded image pixels is encoding.

A38. The system of any of paragraphs A1-A37, wherein the machinelearning algorithm is configured to determine a probabilistic diagnosisof a/the medical condition state of the image input, based on the imageinput and the encoded patient health data.

A39. The system of any of paragraphs A1-A38, wherein the machinelearning algorithm is between 1 and 15 layers deep.

B1. A method of training and preparing a machine learning algorithm formedical condition state determination, the method comprising:

acquiring data from at least one medical procedure, wherein theacquiring data comprises acquiring at least one in situ biological imageof an area of a patient's body and acquiring one or more biologicalspecimens from the area;

labeling the at least one in situ biological image, thereby creating atleast one labeled biological image that indicates respective medicalcondition states shown in each respective biological image;

acquiring patient health data pertaining to the patient from a pluralityof data sources;

aggregating the patient health data acquired from the plurality of datasources into a database;

de-identifying the patient health data in the database; and

training the machine learning algorithm to make medical condition statedeterminations, using the database and the at least one labeledbiological image.

B1.1. The method of paragraph B1, wherein the labeling the at least onein situ biological image comprises labeling a plurality of in situbiological images.

B2. The method of paragraph B1.1, wherein the labeling the plurality ofin situ biological images comprises manual labeling.

B3. The method of any of paragraphs B1.1-B2, wherein the labeling theplurality of in situ biological images comprises semi-supervisedlabeling.

B3.1. The method of any of paragraphs B1.1-B3, wherein the labeling theplurality of in situ biological images comprises manual labeling,followed by semi-supervised labeling.

B3.2. The method of any of paragraphs B1-B3.1, further comprisingperforming auto-labeling, wherein the performing auto-labeling isperformed by the machine learning algorithm.

B4. The method of any of paragraphs B1.1-B3.2, further comprisingassigning a sequential image index to each biological image of theplurality of in situ biological images.

B4.1. The method of any of paragraphs B1-B4, wherein the databasecomprises a text-based database.

B5. The method of any of paragraphs B1-B4.1, wherein the training themachine learning algorithm comprises a plurality of training phases.

B6. The method of paragraph B5, wherein the plurality of training phasescomprises a first phase wherein the labeling is performed based onvisual interpretations of each respective biological image of the atleast one in situ biological image.

B7. The method of paragraph B5 or B6, wherein the plurality of trainingphases comprises a second phase wherein the labeling is performed basedon respective pathology results for a respective biological specimenshown in a respective in situ biological image of the at least one insitu biological image.

B8. The method of any of paragraphs B1-B7, further comprising acquiringpathology results for the one or more biological specimens.

B8.1. The method of any of paragraphs B1-B8, further comprisinganalyzing the at least one biological specimen to determine any presentmedical condition state.

B9. The method of any of paragraphs B5-B.18, wherein a/the plurality oftraining phases comprises a third phase wherein the labeling isperformed based on respective pathology results for the respectivebiological specimen shown in the respective in situ biological image, incombination with bounding boxes to identify an area of interest withinthe respective in situ biological image.

B10. The method of any of paragraphs B5-B9, wherein a/the plurality oftraining phases comprises a fourth phase wherein the labeling isperformed based on respective pathology results for the respectivebiological specimen shown in the respective in situ biological image, incombination with the bounding boxes to identify the area of interestwithin the respective in situ biological image, and wherein the labelingis further based on the patient health data for the patient.

B11. The method of any of paragraphs B5-B10, wherein each phase of theplurality of training phases is implemented serially to train themachine learning algorithm.

B12. The method of any of paragraphs B5-B11, further comprisingobtaining a respective dataset for each respective phase of theplurality of training phases, such that the method comprises using adifferent respective dataset for each respective phase of the pluralityof training phases.

B13. The method of any of paragraphs B1-B12, wherein each biologicalimage of the at least one in situ biological images is at least 578×462pixels.

B14. The method of any of paragraphs B1-B13, wherein each biologicalimage of the at least one in situ biological images is at least 720×486pixels.

B15. The method of any of paragraphs B1-B14, wherein the trainingcomprises training the machine learning algorithm to classify, detect,and/or localize one or more medical condition states in new biologicalimages obtained from new patients.

B16. The method of any of paragraphs B1-B15, further comprisingobtaining a plurality of medical reports for training the machinelearning algorithm.

B17. The method of paragraph B16, further comprising extracting datafrom the plurality of medical reports, wherein the extracting datacomprises extracting images and extracting text data.

B18. The method of paragraph B16 or B17, further comprising performingautomated data de-identification of data extracted from the plurality ofmedical reports.

B19. The method of paragraph B18, wherein the performing automated datade-identification comprises assigning UUIDs, eliminating features,computed age determination, and/or generating a master database and acorresponding de-identified database.

B20. The method of any of paragraphs B1-B19, further comprisingautomated feature elimination of features determined to be unnecessaryby the machine learning algorithm, wherein the automated featureelimination is performed by the machine learning algorithm.

B21. The method of any of paragraphs B1-B20, wherein the machinelearning algorithm comprises a convolutional neural network.

C1. A method of training and preparing a machine learning algorithm formedical condition state determination, the method comprising:

receiving an image input via at least one processing unit, wherein theimage input comprises one or more images from an imaging device, andwherein the at least one processing unit comprises the machine learningalgorithm stored in one or more memories of the at least one processingunit;

receiving patient health data as input, wherein the receiving patienthealth data is performed by the at least one processing unit; and

encoding the patient health data and thereby converting the patienthealth data to encoded patient health data, wherein the encoding thepatient health data and the converting the patient health data isperformed by the at least one processing unit, wherein the machinelearning algorithm is configured to make a medical condition statedetermination based on the image input and the encoded patient healthdata.

C1.1. The method of paragraph C1, further comprising adding the encodedpatient health data to at least one image of the image input, whereinthe adding the encoded patient health data is performed by the at leastone processing unit.

C1.2. The method of paragraph C1.1, wherein the adding the encodedpatient health data comprises embedding the encoded patient health datainto the at least one image of the image input.

C1.3. The method of paragraphs C1.1-C1.2, wherein the adding the encodedpatient health data comprises embedding the encoded patient health datainto a tensor of the machine learning algorithm.

C2. The method of any of paragraphs C1-C1.2, further comprising themethod of any of paragraphs B1-B21.

C3. The method of any of paragraphs C1-C2, further comprising embeddingthe encoded patient health data into the at least one image of the imageinput at or before a time that the machine learning algorithm analyzesthe image input, such that the machine learning algorithm analyzes theimage input together with the encoded patient health data embedded inthe at least one image of the image input.

C3.1. The method of any of paragraphs C1-C3, further comprisingembedding dynamic state information into the at least one image of theimage input at or before a time that the machine learning algorithmanalyzes the image input, such that the machine learning algorithmanalyzes the image input together with the dynamic state informationembedded in the at least one image of the image input.

C3.2. The method of any of paragraphs C1-C3.1, further comprisingembedding the encoded patient health data into the at least one image ofthe image input before the receiving the image input via the at leastone processing unit.

C3.3. The method of any of paragraphs C1-C3.1, further comprisingembedding dynamic state information into the at least one image of theimage input before the receiving the image input via the at least oneprocessing unit.

C4. The method of any of paragraphs C1-C3.3, wherein the machinelearning algorithm comprises a convolutional neural network.

C4.1. The method of any of paragraphs C1-C3.3, wherein the machinelearning algorithm comprises a fully trained convolutional neuralnetwork.

C5. The method of any of paragraphs C1-C4.1, wherein the machinelearning algorithm comprises a transfer learning convolutional neuralnetwork.

C6. The method of any of paragraphs C1-C5, wherein the encoding thepatient health data comprises performing one-hot encoding conversion.

C6.1. The method of any of paragraphs C1-C6, wherein the encoding thepatient health data comprises performing data dictionary encoding.

C6.2. The method of any of paragraphs C1-C6.1, further comprisingencoding dynamic state information via one-hot encoding conversion.

C6.3. The method of any of paragraphs C1-C6.2, further comprisingencoding dynamic state information via data dictionary encoding.

C7. The method of any of paragraphs C1-C6.3, further comprising imageencoding the encoded patient health data, wherein the image encoding isperformed by the at least one processing unit, and wherein the imageencoding is performed after the encoding the patient health data.

C7.1. The method of paragraph C7, wherein the image encoding comprisesadding, overlaying, and/or appending image pixels to the image input.

C7.2. The method of any of paragraphs C1-C7.1, further comprising imageencoding the dynamic state information, wherein the image encoding isperformed by the at least one processing unit, and wherein the imageencoding is performed after encoding the dynamic state information.

C7.3. The method of paragraph C7.2, wherein the image encoding comprisesadding, overlaying, and/or appending image pixels to the image input.

C8. The method of any of paragraphs C1-C7.3, further comprisingperforming a flattening operation to flatten the encoded patient healthdata and/or the at least one image, wherein the performing theflattening operation is performed by the at least one processing unit.

C8.1. The method of paragraph C8, wherein the performing the flatteningoperation further comprises flattening the encoded dynamic stateinformation.

C8.2. The method of any of paragraphs C1-C8.1, further comprisingperforming a concatenating operation to concatenate the encoded patienthealth data and/or the at least one image, wherein the performing theconcatenating operation is performed by the at least one processingunit.

C8.3. The method of paragraph C8.2, wherein the performing theconcatenating operation further comprises concatenating the encodeddynamic state information.

C8.4. The method of paragraph C8.2 or C8.3, wherein the performing theconcatenating is performed prior to or within a multi-layer perceptronportion of the machine learning algorithm.

C9. The method of any of paragraphs C1-C8.4, further comprisingdisplaying the encoded patient health data within a consistent region ofan output image, wherein the displaying is performed by the at least oneprocessing unit.

C9.1. The method of paragraph C9, wherein the displaying the encodedpatient health data comprises displaying the encoded patient health datavia labels and/or icons.

C9.2. The method of paragraph C9 or C9.1, wherein the displaying theencoded patient health data comprises displaying real time patienthealth history data to a user.

C10. The method of any of paragraphs C1-C9.2, comprising adding theencoded patient health data, wherein the adding the encoded patienthealth data comprises embedding the encoded patient health data withina/the consistent region of the at least one image.

C11. The method of any of paragraphs C1-C10, wherein the encoding thepatient health data comprises converting the patient health data to aplurality of collections of coded image pixels that are added to,appended to, overlaid on, and/or embedded in the at least one image ofthe image input.

C12. The method of any of paragraphs C1-C11, further comprising adding,appending, and/or overlaying the encoded patient health data to theimage input as a vector and/or a data tensor, wherein the adding,appending, and/or overlaying is performed by the at least one processingunit.

C12.1. The method of any of paragraphs C1-C12, further comprisingadding, appending, and/or overlaying encoded dynamic state informationto the image input as a/the vector and/or a/the data tensor, wherein theadding, appending, and/or overlaying is performed by the at least oneprocessing unit.

C12.2. The method of any of paragraphs C1-C12.1, further comprisingadding, appending, and/or overlaying the encoded patient health data toa data tensor, wherein the adding, appending, and/or overlaying isperformed by the at least one processing unit.

C13. The method of paragraph C12 or C12.1, wherein the adding,appending, and/or overlaying the encoded patient health data and/or theencoded dynamic state information is performed after flattening and/orconcatenating the image input and/or the encoded patient health data.

C14. The method of any of paragraphs C1-C13, wherein the encoding thepatient health data comprises selectively representing the patienthealth data and/or dynamic state information as a/the tensor and/ora/the plurality of collections of coded image pixels.

C15. The method of any of paragraphs C1-C14, further comprisinginputting the encoded patient health data into a fully connected networkportion of the machine learning algorithm.

C15.1. The method of any of paragraphs C1-C15, further comprisinginputting encoded dynamic state information into a/the fully connectednetwork portion of the machine learning algorithm.

C16. The method of any of paragraphs C1-C15.1, wherein the making themedical condition state determination is performed in real-time.

C16.1. The method of any of paragraphs C1-C16, wherein the making themedical condition state determination is automated.

C17. The method of any of paragraphs C1-C16.1, further comprising:

collecting and/or retrieving the patient health data; and

delivering the patient health data to the at least one processing unit.

C18. The method of paragraph C17, wherein the collecting and/orretrieving the patient health data is performed in real-time.

C18.1. The method of any of paragraphs C1-C18, wherein the patienthealth data is collected or provided before the receiving the imageinput.

C19. The method of any of paragraphs C1-C18.1, wherein the patienthealth data comprises survey question answers, static data, electronichealth records, electronic medical records, demographic information,medications, drug use, smoking history, computed risk predictors, bloodwork, prior procedural results, and/or risk factors.

C19.1. The method of any of paragraphs C1-C19, wherein the patienthealth data comprises body mass index (BMI), physical activity,cigarette smoking history, alcohol usage, family history, presence ofinflammatory bowel disease, current hormone therapy (e.g.,postmenopausal hormone therapy), former hormone therapy (e.g.,postmenopausal hormone therapy), aspirin usage, nonsteroidalanti-inflammatory drugs (NSAIDs) usage, consumption of processed and/orred meat, fruit and vegetable consumption levels, demographicinformation, medications, drug usage, diet type and quality, dietary fatintake, weight, height, age, race, presence of other illnesses,biological markers, INR/PTT/PT/platelets/bleed time, previous endoscopyresults, previous CT scan results, previous angiogram information,previous MRI results, and/or previous sonography data.

C20. The method of any of paragraphs C1-C19, wherein the making themedical condition state determination comprises detecting a medicalcondition state based on the one or more images and the patient healthdata.

C20.1. The method of paragraph C20, further comprising displaying themedical condition state.

C20.2. The method of any of paragraphs C1-C20.1, further comprisingdisplaying the patient health data.

C21. The method of any of paragraphs C1-C20.2, wherein the making themedical condition state determination comprises classifying the medicalcondition state based on the one or more images and the patient healthdata.

C22. The method of any of paragraphs C1-C21, wherein the making themedical condition state determination comprises localizing one or moremedical condition states based on the one or more images and the patienthealth data.

C23. The method of any of paragraphs C1-C22, further comprisingde-identifying the patient health data, wherein the de-identifying thepatient health data is performed by the at least one processing unit.

C24. The method of any of paragraphs C1-C23, further comprisingreceiving manual labeling information for the image input, wherein thereceiving the manual labeling information is performed by the at leastone processing unit.

C25. The method of any of paragraphs C1-C24, further comprisingreceiving semi-supervised labeling information for the image input,wherein the receiving the semi-supervised labeling information isperformed by the at least one processing unit.

C26. The method of any of paragraphs C1-C25, further comprisingperforming a labeling feedback loop comprising the semi-supervisedlabeling information, wherein the performing the labeling feedback loopis partially performed by the at least one processing unit.

C26.1. The method of paragraph C26, wherein the performing the labelingfeedback loop improves training efficiency of the machine learningalgorithm.

C26.2. The method of paragraph C26 or C26.1, wherein the performing thelabeling feedback loop enables training of the machine learningalgorithm using large scale data and/or big data.

C27. The method of any of paragraphs C1-C26.1, wherein the machinelearning algorithm comprises a custom convolutional neural network.

C28. The method of any of paragraphs C1-C27, wherein the machinelearning algorithm is informed and trained using both image trainingdata and patient health training data that is encoded such that it maybe added to, appended to, overlaid on, and/or embedded in the imagetraining data.

C29. The method of any of paragraphs C1-C28, further comprisingproducing an/the output image that comprises visual output for medicalcondition state determination that is augmented with the encoded patienthealth data, wherein the producing the output image is performed by themachine learning algorithm.

C29.1. The method of any of paragraphs C1-C28, further comprisingproducing an/the output image that comprises visual output for medicalcondition state determination that is augmented with the encoded patienthealth data, wherein the producing the output image is performed by anencoding algorithm of the at least one processing unit.

C30. The method of any of paragraphs C1-C29.1, wherein the output imagecomprises the at least one image of the image input and the encodedpatient health data.

C31. The method of any of paragraphs C1-C30, wherein the output imagefurther comprises the medical condition state determination determinedby the machine learning algorithm.

C32. The method of any of paragraphs C1-C31, further comprisingreceiving dynamic state information of a patient via an apparatus,wherein the apparatus is configured to deliver the dynamic stateinformation to the at least one processing unit as an additional inputsuch that the receiving the dynamic state information is performed bythe at least one processing unit.

C32.1 The method of paragraph C32, wherein the dynamic state informationis sensor-derived data obtained in real-time during a medical imagingprocedure that produces the image input.

C32.2. The method of any of paragraphs C32-C32.1, wherein the dynamicstate information comprises heart rate, blood pressure, compensatedheart rate, anesthetics, telemetry, saline or other fluids used, oxygensaturation, end tidal carbon dioxide (capnography), and/or activity indistal extremities.

C32.3. The method of any of paragraphs C32-C32.2, wherein the apparatuscomprises one or more motion sensors, one or more accelerometers, and/orone or more video cameras configured for motion detection.

C33. The method of any of paragraphs C32-C32.3, wherein the dynamicstate information comprises information regarding a patient's cardiaccycle and/or breathing cycle, and wherein the image input is temporallyannotated such that each of the one or more images from the imagingdevice may be matched with a phase of the patient's cardiac cycle and/orbreathing cycle.

C33.1. The method of any of paragraphs C32-C33, further comprisingencoding the dynamic state information, wherein the encoding the dynamicstate information is performed by the at least one processing unit.

C34. The method of any of paragraphs C1-C33.1, further comprising onlinelearning performed by updating the machine learning algorithm usingneural network weights, such that the machine learning algorithmcontinues to learn as it receives additional image input and patienthealth data.

C35. The method of any of paragraphs C1-C34, wherein the imaging devicecomprises a sonography device, an x-ray device, a computed tomography(CT) scanning device, a magnetic resonance imaging (MRI) device, apositron emission tomography (PET) device, a retinal camera, adermatoscope, a radiograph device, a mammography device, an endoscope, acolonoscopy device, an esophagogastroduodenoscopy device, a bronchoscopydevice, and/or a capsule endoscopy device.

C36. The method of any of paragraphs C1-C35, further comprisingprocessing and interpreting the encoded patient health data, wherein theencoded patient health data is embedded in the at least one image of theimage input as a/the plurality of collections of coded image pixels.

C37. The method of paragraph C36, wherein the plurality of collectionsof coded image pixels comprises a respective collection of coded imagepixels for each respective type or category of encoded patient healthdata.

C38. The method of paragraph C36 or C37, wherein the collections ofcoded image pixels are arranged in a row, a column, and/or an array onthe at least one image.

C39. The method of any of paragraphs C36-C38, wherein the collections ofcoded image pixels are positioned together within a given region of theat least one image.

C40. The method of any of paragraphs C3-C39, wherein a respective shadeof each respective collection of coded image pixels represents arelative value of the respective encoded patient health data encoded inthe respective collection of coded image pixels.

C41. The method of any of paragraphs C36-C40, wherein the plurality ofcollections of coded image pixels comprises a plurality ofgrayscale-coded image pixels.

C42. The method of any of paragraphs C36-C41, wherein the plurality ofcollections of coded image pixels comprises a plurality of color-codedimage pixels.

C43. The method of any of paragraphs C36-C42, further comprisingdisplaying a respective icon for each respective collection of codedimage pixels in an/the output image, wherein the respective icon isconfigured to indicate what the respective collection of coded imagepixels is encoding, wherein the displaying the respective icon isperformed by the machine learning algorithm.

C43.1. The method of any of paragraphs C36-C42, further comprisingdisplaying a respective icon for each respective collection of codedimage pixels in an/the output image, wherein the respective icon isconfigured to indicate what the respective collection of coded imagepixels is encoding, wherein the displaying the respective icon isperformed by an/the encoding algorithm of the at least one processingunit.

C44. The method of any of paragraphs C1-C43.1, further comprisingdetermining a probabilistic diagnosis of the medical condition state ofthe image input, based on the image input and the encoded patient healthdata, wherein the determining the probabilistic diagnosis is performedby the machine learning algorithm.

C45. The method of any of paragraphs C1-C44, further comprising makinga/the medical condition state determination, via the machine learningalgorithm, based on the image input and the encoded patient health data.

C46. The method of any of paragraphs C1-C45, further comprisingproducing an/the output image, via the machine learning algorithm,wherein the output image comprises the at least one image of the imageinput and the encoded patient health data.

C46.1. The method of any of paragraphs C1-C45, further comprisingproducing an output image, via an/the encoding algorithm of the at leastone processing unit, wherein the output image comprises the at least oneimage of the image input and the encoded patient health data.

C47. The method of paragraph C46 or C46.1, wherein the output imagefurther comprises the medical condition state determination determinedby the machine learning algorithm.

C48. The method of any of paragraph C46-C47, wherein the output imagefurther comprises current polyp count in real-time during a/the medicalimaging procedure, predicted distance of travel of the imaging devicewithin a patient's body, upcoming landmarks within the patient's body,information from previously performed medical procedures,recommendations on anesthesia, probability rates of cancer in a givenarea of the patient's body, a live probability of finding a polyp, alive probability of the pathology of a polyp, most recent medication thepatient received, a predicated date for subsequent procedures, apredicted model of an organ of the patient being imaged, and/or summaryinformation regarding the medical imaging procedure.

D1. A method of training and preparing a machine learning algorithm formedical condition state determination, the method comprising:

programming at least one processing unit to receive an image input,wherein the image input comprises one or more images from an imagingdevice, and wherein the at least one processing unit comprises a machinelearning algorithm;

programming the at least one processing unit to receive patient healthdata as input;

programming the at least one processing unit to encode the patienthealth data and thereby convert the patient health data to encodedpatient health data; and

programming the at least one processing unit to embed the encodedpatient health data into at least one image of the image input, whereinthe at least one processing unit is configured to make a medicalcondition state determination, via the machine learning algorithm, basedon the image input and the encoded patient health data.

D2. The method of paragraph D1, comprising programming the at least oneprocessing unit to perform the method of any of paragraphs C1-C48.

D3. The method of paragraph D1 or D2, wherein the machine learningalgorithm comprises a convolutional neural network.

E1. A machine learning algorithm-implemented method for making a medicalcondition state determination, the method comprising:

accessing an image input, wherein the image input comprises one or moreimages from an imaging device used to perform a medical imagingprocedure on a patient;

accessing patient health data;

causing a machine learning algorithm to analyze the image input and thepatient health data together to make the medical condition statedetermination, wherein the machine learning algorithm is configured toencode the patient health data to convert the patient health data toencoded patient health data, and wherein the machine learning algorithmis further configured to embed the encoded patient health data into atleast one image of the image input; and

accessing analysis results produced by the machine learning algorithm,wherein the analysis results comprise the medical condition statedetermination, and wherein the analysis results further comprise avisual representation of the encoded patient health data viewable on theanalysis results.

E2. The method of paragraph E1, further comprising the method of any ofparagraphs B1-B21.

E3. The method of any of paragraphs E1-E2, further comprising the methodof any of paragraphs C1-C48.

E4. The method of any of paragraphs E1-E3, wherein the imaging devicecomprises a sonography device, an x-ray device, a computed tomography(CT) scanning device, a magnetic resonance imaging (MRI) device, apositron emission tomography (PET) device, a retinal camera, adermatoscope, a radiograph device, a mammography device, an endoscope, acolonoscopy device, an esophagogastroduodenoscopy device, a bronchoscopydevice, and/or a capsule endoscopy device.

E5. The method of any of paragraphs E1-E4, wherein the method utilizesthe system of any of paragraphs A1-A39.

E6. The method of any of paragraphs E1-E5, further comprisingdetermining dynamic state information of the patient using an apparatus,wherein the apparatus is configured to deliver the dynamic stateinformation to the machine learning algorithm as an additional input.

E7. The method of paragraph E6, wherein the dynamic state information issensor-derived data obtained in real-time during the medical imagingprocedure.

E8. The method of paragraph E6 or E7, wherein the dynamic stateinformation comprises information regarding a/the patient's cardiaccycle and/or breathing cycle.

E9. The method of any of paragraphs E6-E8, wherein the dynamic stateinformation comprises information regarding a/the patient's cardiaccycle and/or breathing cycle, and wherein the image input is temporallyannotated such that each of the one or more images from the imagingdevice may be matched with a phase of the patient's cardiac cycle and/orbreathing cycle.

E10. The method of any of paragraphs E1-E9, further comprising matchingthe one or more images from the imaging device with cardiac cycleinformation of the patient, wherein the cardiac cycle information isdetermined dynamically during the medical imaging procedure, such thateach of the one or more images from the imaging device may be matchedwith a phase of the patient's cardiac cycle.

E10.1. The method of any of paragraphs E1-E10, further comprisingmatching the one or more images from the imaging device with breathingcycle information of the patient, wherein the breathing cycleinformation is determined dynamically during the medical imagingprocedure, such that each of the one or more images from the imagingdevice may be matched with a phase of the patient's breathing cycle.

E11. The method of any of paragraphs E7-E10.1, wherein the dynamic stateinformation comprises heart rate, blood pressure, compensated heartrate, anesthetics, telemetry, saline used, other fluids used, oxygensaturation, end tidal carbon dioxide (capnography), and/or activity indistal extremities.

E12. The method of any of paragraphs E6-E11, wherein the apparatuscomprises one or more motion sensors, one or more accelerometers, and/orone or more video cameras configured for motion detection.

E13. The method of any of paragraphs E1-E12, wherein the machinelearning algorithm comprises a convolutional neural network.

E14. The method of any of paragraphs E1-E13, wherein the accessing thepatient health data comprises collecting the patient health data inreal-time.

E15. The method of any of paragraphs E1-E14, wherein the accessing thepatient health data comprises retrieving the patient health data,wherein the patient health data was collected or provided before theaccessing the image input.

F1. The use of the system of any of paragraphs A1-A39 to make a medicalcondition state determination.

As used herein, the terms “selective” and “selectively,” when modifyingan action, movement, configuration, or other activity of one or morecomponents or characteristics of an apparatus, mean that the specificaction, movement, configuration, or other activity is a direct orindirect result of dynamic processes and/or user manipulation of anaspect of, or one or more components of, the apparatus. The terms“selective” and “selectively” thus may characterize an activity that isa direct or indirect result of user manipulation of an aspect of, or oneor more components of, the apparatus, or may characterize a process thatoccurs automatically, such as via the mechanisms disclosed herein.

As used herein, the terms “adapted” and “configured” mean that theelement, component, or other subject matter is designed and/or intendedto perform a given function. Thus, the use of the terms “adapted” and“configured” should not be construed to mean that a given element,component, or other subject matter is simply “capable of” performing agiven function but that the element, component, and/or other subjectmatter is specifically selected, created, implemented, utilized,programmed, and/or designed for the purpose of performing the function.It is also within the scope of the present disclosure that elements,components, and/or other recited subject matter that is recited as beingadapted to perform a particular function may additionally oralternatively be described as being configured to perform that function,and vice versa. Similarly, subject matter that is recited as beingconfigured to perform a particular function may additionally oralternatively be described as being operative to perform that function.

As used herein, the phrase “at least one,” in reference to a list of oneor more entities should be understood to mean at least one entityselected from any one or more of the entities in the list of entities,but not necessarily including at least one of each and every entityspecifically listed within the list of entities and not excluding anycombinations of entities in the list of entities. This definition alsoallows that entities may optionally be present other than the entitiesspecifically identified within the list of entities to which the phrase“at least one” refers, whether related or unrelated to those entitiesspecifically identified. Thus, as a non-limiting example, “at least oneof A and B” (or, equivalently, “at least one of A or B,” or,equivalently “at least one of A and/or B”) may refer, in one example, toat least one, optionally including more than one, A, with no B present(and optionally including entities other than B); in another example, toat least one, optionally including more than one, B, with no A present(and optionally including entities other than A); in yet anotherexample, to at least one, optionally including more than one, A, and atleast one, optionally including more than one, B (and optionallyincluding other entities). In other words, the phrases “at least one,”“one or more,” and “and/or” are open-ended expressions that are bothconjunctive and disjunctive in operation. For example, each of theexpressions “at least one of A, B, and C,” “at least one of A, B, or C,”“one or more of A, B, and C,” “one or more of A, B, or C” and “A, B,and/or C” may mean A alone, B alone, C alone, A and B together, A and Ctogether, B and C together, or A, B, and C together, and optionally anyof the above in combination with at least one other entity.

The various disclosed elements of apparatuses and steps of methodsdisclosed herein are not required to all apparatuses and methodsaccording to the present disclosure, and the present disclosure includesall novel and non-obvious combinations and subcombinations of thevarious elements and steps disclosed herein. Moreover, one or more ofthe various elements and steps disclosed herein may define independentinventive subject matter that is separate and apart from the whole of adisclosed apparatus or method. Accordingly, such inventive subjectmatter is not required to be associated with the specific apparatusesand methods that are expressly disclosed herein, and such inventivesubject matter may find utility in apparatuses and/or methods that arenot expressly disclosed herein.

As used herein, the phrase, “for example,” the phrase, “as an example,”and/or simply the term “example,” when used with reference to one ormore components, features, details, structures, examples, and/or methodsaccording to the present disclosure, are intended to convey that thedescribed component, feature, detail, structure, example, and/or methodis an illustrative, non-exclusive example of components, features,details, structures, examples, and/or methods according to the presentdisclosure. Thus, the described component, feature, detail, structure,example, and/or method is not intended to be limiting, required, orexclusive/exhaustive; and other components, features, details,structures, examples, and/or methods, including structurally and/orfunctionally similar and/or equivalent components, features, details,structures, examples, and/or methods, are also within the scope of thepresent disclosure.

1. A system for preparing, training, and deploying a machine learningalgorithm for medical condition state determination, the systemcomprising: at least one processing unit comprising the machine learningalgorithm, wherein the machine learning algorithm is stored in one ormore memories of the at least one processing unit, wherein the at leastone processing unit is programmed to: receive an image input from animaging device, wherein the image input comprises one or more imagesobtained by the imaging device; receive patient health data as input;encode the patient health data to convert the patient health data toencoded patient health data; and transmit the encoded patient healthdata into the machine learning algorithm, wherein the system isconfigured to make a medical condition state determination based on theimage input and the encoded patient health data, via the machinelearning algorithm, and wherein the system is further configured toprovide visual output for the medical condition state determination viaa display device, wherein the visual output is augmented with thepatient health data.
 2. The system according to claim 1, wherein thesystem is configured such that the encoded patient health data isembedded into at least one image of the image input at or before a timethat the machine learning algorithm analyzes the image input, such thatthe machine learning algorithm analyzes the image input together withthe encoded patient health data embedded in the at least one image ofthe image input.
 3. The system according to claim 1, further comprisingthe imaging device, wherein the imaging device is configured to producethe one or more images, and wherein the imaging device comprises one ormore selected from the group consisting of a sonography device, an x-raydevice, a computed tomography (CT) scanning device, a magnetic resonanceimaging (MRI) device, a positron emission tomography (PET) device, aretinal camera, a dermatoscope, a radiograph device, a mammographydevice, an endoscope, a colonoscopy device, anesophagogastroduodenoscopy device, a bronchoscopy device, aphotoacoustic endoscopy device, an electro-optical sensor, a NBI (NarrowBand Imaging) colonoscopy device, a white light endoscopy device, achromoendoscopy device, and a capsule endoscopy device.
 4. The systemaccording to claim 1, wherein the at least one processing unit comprisesan encoding algorithm configured to produce an output image thatcomprises at least one image of the image input and the encoded patienthealth data, wherein the system is configured to display the encodedpatient health data within a region of the at least one image, andwherein the encoded patient health data comprises a plurality ofcollections of coded image pixels that are added to the image input. 5.The system according to claim 4, wherein the at least one processingunit is further programmed to add encoded dynamic state information tothe image input as a tensor.
 6. The system according to claim 1, whereinthe at least one processing unit is further programmed to add theencoded patient health data to the image input as a tensor.
 7. Thesystem according to claim 6, wherein the at least one processing unit isfurther programmed to add encoded dynamic state information to the imageinput as the tensor.
 8. The system according to claim 1, wherein themachine learning algorithm is configured to receive the encoded patienthealth data into a fully connected network portion of the machinelearning algorithm.
 9. The system according to claim 1, wherein thesystem is configured to perform real-time, medical condition statedetermination.
 10. The system according to claim 1, further comprising acomputing device configured for one or both of (i) collecting thepatient health data in real-time and (ii) retrieving the patient healthdata in real-time from a database, wherein the computing device isfurther configured to deliver the patient health data to the at leastone processing unit.
 11. The system according to claim 1, wherein themachine learning algorithm is configured to detect, classify, andlocalize one or more medical condition states based on the one or moreimages and the patient health data.
 12. The system according to claim 1,wherein the system is configured to accept manual labeling orsemi-supervised labeling by utilizing the machine learned model toassign a plurality of initial labels, followed by manual verificationfor at least a portion of the initial labels.
 13. The system accordingto claim 1, wherein the machine learning algorithm is trained using oneor more selected from the group consisting of unsupervised learning,semi-supervised learning, and supervised learning.
 14. The systemaccording to claim 1, wherein the patient health data comprisesinformation regarding a patient's cardiac cycle and/or breathing cycle,and wherein the image input is temporally annotated such that each ofthe one or more images from the imaging device may be matched with aphase of the patient's cardiac cycle and/or breathing cycle.
 15. Thesystem according to claim 1, further comprising an apparatus fordetermining dynamic state information of a patient, wherein theapparatus is configured to deliver the dynamic state information to theat least one processing unit as an additional input, and wherein thedynamic state information comprises one or more selected from the groupconsisting of heart rate, blood pressure, compensated heart rate,anesthetics, telemetry, saline used, other fluids used, oxygensaturation, end tidal carbon dioxide (capnography), current medications,and activity in distal extremities.
 16. The system according to claim15, wherein the system is configured such that the dynamic stateinformation is embedded into at least one image of the image input at orbefore a time that the machine learning algorithm analyzes the imageinput, such that the machine learning algorithm analyzes the image inputtogether with the dynamic state information embedded in the at least oneimage of the image input.
 17. The system according to claim 1, whereinthe machine learning algorithm comprises a convolutional neural network.18. The system according to claim 1, wherein the system is configured toautomatically generate a report for a patient that includes a summary ofthe medical condition state determination, along with billinginformation for the procedure.